/root/miniconda3/envs/train_vlm/bin/python3.10: can't open file '/root/ai-project/Vintern/internvl/train/internvl_chat_finetune.py': [Errno 2] No such file or directory
E0703 09:24:57.109000 138100931848000 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 2) local_rank: 0 (pid: 31203) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:24:57
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 2 (pid: 31203)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/bin/python3.10: can't open file '/root/ai-project/Vintern/internvl/train/internvl_chat_finetune.py': [Errno 2] No such file or directory
E0703 09:25:53.714000 130563156473664 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 2) local_rank: 0 (pid: 31222) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:25:53
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 2 (pid: 31222)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/bin/python3.10: can't open file '/root/ai-project/Vintern/internvl_chatinternvl/train/internvl_chat_finetune.py': [Errno 2] No such file or directory
E0703 09:30:47.026000 128480840795968 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 2) local_rank: 0 (pid: 31255) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chatinternvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:30:47
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 2 (pid: 31255)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.

0it [00:00, ?it/s]
0it [00:00, ?it/s]
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
[2026-07-03 09:31:35,988] [WARNING] [real_accelerator.py:199:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
Traceback (most recent call last):
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 32, in <module>
    from internvl.train.dataset import (ConcatDataset, TCSLoader,
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/dataset.py", line 10, in <module>
    import cv2
ModuleNotFoundError: No module named 'cv2'
E0703 09:31:46.332000 132150586988352 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 31273) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:31:46
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 31273)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
[2026-07-03 09:32:20,885] [WARNING] [real_accelerator.py:199:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
Traceback (most recent call last):
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 32, in <module>
    from internvl.train.dataset import (ConcatDataset, TCSLoader,
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/dataset.py", line 11, in <module>
    import imageio
ModuleNotFoundError: No module named 'imageio'
E0703 09:32:32.354000 139803546371904 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 31354) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:32:32
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 31354)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
[2026-07-03 09:33:03,430] [WARNING] [real_accelerator.py:199:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Traceback (most recent call last):
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 847, in <module>
    main()
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 608, in main
    init_dist(launcher=launcher, backend='nccl')
  File "/root/ai-project/Vintern/internvl_chat/internvl/dist_utils.py", line 36, in init_dist
    _init_dist_pytorch(backend, **kwargs)
  File "/root/ai-project/Vintern/internvl_chat/internvl/dist_utils.py", line 49, in _init_dist_pytorch
    torch.cuda.set_device(rank % num_gpus)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py", line 420, in set_device
    torch._C._cuda_setDevice(device)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py", line 314, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination
E0703 09:33:12.348000 129426906138432 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 31441) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:33:12
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 31441)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/03/2026 09:42:04 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/03/2026 09:42:04 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=16,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul03_09-42-04_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=10,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=1.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/03/2026 09:42:04 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:42:04,130 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:42:04,131 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:42:04,131 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:42:04,131 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:42:04,131 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:42:04,131 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-03 09:42:04,523 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/03/2026 09:42:04 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-03 09:42:04,532 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-03 09:42:04,534 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/03/2026 09:42:04 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-03 09:42:04,535 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-03 09:42:05,383 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-03 09:42:05,384 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-03 09:42:05,441 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-03 09:42:16,406 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-03 09:42:16,407 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-03 09:42:16,425 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-03 09:42:16,426 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/03/2026 09:42:16 - INFO - __main__ - Finished
07/03/2026 09:42:16 - INFO - __main__ - model.config.force_image_size: 448
07/03/2026 09:42:16 - INFO - __main__ - data_args.force_image_size: 448
07/03/2026 09:42:16 - INFO - __main__ - model.config.vision_config.image_size: 448
[rank0]: Traceback (most recent call last):
[rank0]:   File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 847, in <module>
[rank0]:     main()
[rank0]:   File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 765, in main
[rank0]:     train_dataset = build_datasets(
[rank0]:   File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 562, in build_datasets
[rank0]:     ds_collections = json.loads(open(data_args.meta_path).read())
[rank0]:   File "/root/miniconda3/envs/train_vlm/lib/python3.10/json/__init__.py", line 346, in loads
[rank0]:     return _default_decoder.decode(s)
[rank0]:   File "/root/miniconda3/envs/train_vlm/lib/python3.10/json/decoder.py", line 337, in decode
[rank0]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[rank0]:   File "/root/miniconda3/envs/train_vlm/lib/python3.10/json/decoder.py", line 355, in raw_decode
[rank0]:     raise JSONDecodeError("Expecting value", s, err.value) from None
[rank0]: json.decoder.JSONDecodeError: Expecting value: line 5 column 21 (char 163)
E0703 09:42:18.803000 138079627822912 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 31676) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:42:18
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 31676)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/03/2026 09:45:58 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/03/2026 09:45:58 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=16,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul03_09-45-58_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=10,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=1.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/03/2026 09:45:58 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:45:58,889 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:45:58,889 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:45:58,889 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:45:58,889 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:45:58,889 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:45:58,889 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-03 09:45:59,893 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/03/2026 09:45:59 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-03 09:45:59,928 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-03 09:45:59,931 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/03/2026 09:45:59 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-03 09:45:59,933 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-03 09:45:59,962 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-03 09:45:59,964 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-03 09:46:00,035 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-03 09:46:02,099 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-03 09:46:02,099 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-03 09:46:02,104 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-03 09:46:02,104 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/03/2026 09:46:02 - INFO - __main__ - Finished
07/03/2026 09:46:02 - INFO - __main__ - model.config.force_image_size: 448
07/03/2026 09:46:02 - INFO - __main__ - data_args.force_image_size: 448
07/03/2026 09:46:02 - INFO - __main__ - model.config.vision_config.image_size: 448
[rank0]: Traceback (most recent call last):
[rank0]:   File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 847, in <module>
[rank0]:     main()
[rank0]:   File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 765, in main
[rank0]:     train_dataset = build_datasets(
[rank0]:   File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 562, in build_datasets
[rank0]:     ds_collections = json.loads(open(data_args.meta_path).read())
[rank0]:   File "/root/miniconda3/envs/train_vlm/lib/python3.10/json/__init__.py", line 346, in loads
[rank0]:     return _default_decoder.decode(s)
[rank0]:   File "/root/miniconda3/envs/train_vlm/lib/python3.10/json/decoder.py", line 337, in decode
[rank0]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[rank0]:   File "/root/miniconda3/envs/train_vlm/lib/python3.10/json/decoder.py", line 355, in raw_decode
[rank0]:     raise JSONDecodeError("Expecting value", s, err.value) from None
[rank0]: json.decoder.JSONDecodeError: Expecting value: line 5 column 21 (char 175)
E0703 09:46:03.625000 140571859314496 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 31802) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:46:03
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 31802)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Traceback (most recent call last):
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 18, in <module>
    from internvl.dist_utils import init_dist
ModuleNotFoundError: No module named 'internvl'
E0703 09:51:46.399000 129718937388864 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 31938) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_09:51:46
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 31938)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/03/2026 09:52:26 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/03/2026 09:52:26 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=16,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul03_09-52-26_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=10,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=1.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/03/2026 09:52:26 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:52:26,847 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:52:26,847 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:52:26,847 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:52:26,847 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:52:26,847 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:52:26,847 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-03 09:52:27,597 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/03/2026 09:52:27 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-03 09:52:27,606 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-03 09:52:27,608 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/03/2026 09:52:27 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-03 09:52:27,609 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-03 09:52:27,630 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-03 09:52:27,632 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-03 09:52:27,689 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-03 09:52:28,772 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-03 09:52:28,772 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-03 09:52:28,777 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-03 09:52:28,777 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/03/2026 09:52:28 - INFO - __main__ - Finished
07/03/2026 09:52:28 - INFO - __main__ - model.config.force_image_size: 448
07/03/2026 09:52:28 - INFO - __main__ - data_args.force_image_size: 448
07/03/2026 09:52:28 - INFO - __main__ - model.config.vision_config.image_size: 448
07/03/2026 09:52:28 - INFO - __main__ - max_dynamic_patch is set to 12 according to the meta file
07/03/2026 09:52:28 - INFO - __main__ - [Dataset] num_image_token: 256
07/03/2026 09:52:28 - INFO - __main__ - [Dataset] dynamic_image_size: True
07/03/2026 09:52:28 - INFO - __main__ - [Dataset] use_thumbnail: True
07/03/2026 09:52:28 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 12
07/03/2026 09:52:28 - INFO - __main__ - Formatting inputs...Skip in lazy mode
07/03/2026 09:52:29 - INFO - __main__ - Add dataset: vi-envelope-vqa with length: 8843
trainable params: 8,798,208 || all params: 638,496,128 || trainable%: 1.3779579255334184
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight
07/03/2026 09:52:30 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight
07/03/2026 09:52:30 - INFO - __main__ - mlp1.0.weight
07/03/2026 09:52:30 - INFO - __main__ - mlp1.0.bias
07/03/2026 09:52:30 - INFO - __main__ - mlp1.1.weight
07/03/2026 09:52:30 - INFO - __main__ - mlp1.1.bias
07/03/2026 09:52:30 - INFO - __main__ - mlp1.3.weight
07/03/2026 09:52:30 - INFO - __main__ - mlp1.3.bias
[INFO|trainer.py:641] 2026-07-03 09:52:30,155 >> Using auto half precision backend
[INFO|trainer.py:2078] 2026-07-03 09:53:27,067 >> ***** Running training *****
[INFO|trainer.py:2079] 2026-07-03 09:53:27,067 >>   Num examples = 8,843
[INFO|trainer.py:2080] 2026-07-03 09:53:27,067 >>   Num Epochs = 1
[INFO|trainer.py:2081] 2026-07-03 09:53:27,067 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2084] 2026-07-03 09:53:27,068 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:2085] 2026-07-03 09:53:27,068 >>   Gradient Accumulation steps = 16
[INFO|trainer.py:2086] 2026-07-03 09:53:27,068 >>   Total optimization steps = 552
[INFO|trainer.py:2087] 2026-07-03 09:53:27,073 >>   Number of trainable parameters = 13,281,024
[INFO|integration_utils.py:723] 2026-07-03 09:53:27,078 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
wandb: Currently logged in as: hhung2662000 (hhung26062000) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run u11ur5ga
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in /root/ai-project/Vintern/internvl_chat/wandb/run-20260703_095327-u11ur5ga
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run vintern_lora_card
wandb: ⭐️ View project at https://wandb.ai/hhung26062000/huggingface
wandb: 🚀 View run at https://wandb.ai/hhung26062000/huggingface/runs/u11ur5ga

  0%|          | 0/552 [00:00<?, ?it/s]/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/03/2026 09:55:15 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/03/2026 09:55:15 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=128,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul03_09-55-15_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=10,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=1.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/03/2026 09:55:15 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:55:15,932 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:55:15,932 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:55:15,932 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:55:15,932 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:55:15,932 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 09:55:15,932 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-03 09:55:16,648 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/03/2026 09:55:16 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-03 09:55:16,658 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-03 09:55:16,660 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/03/2026 09:55:16 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-03 09:55:16,660 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-03 09:55:16,684 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-03 09:55:16,685 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-03 09:55:16,743 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-03 09:55:18,133 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-03 09:55:18,134 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-03 09:55:18,144 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-03 09:55:18,145 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/03/2026 09:55:18 - INFO - __main__ - Finished
07/03/2026 09:55:18 - INFO - __main__ - model.config.force_image_size: 448
07/03/2026 09:55:18 - INFO - __main__ - data_args.force_image_size: 448
07/03/2026 09:55:18 - INFO - __main__ - model.config.vision_config.image_size: 448
07/03/2026 09:55:18 - INFO - __main__ - max_dynamic_patch is set to 12 according to the meta file
07/03/2026 09:55:18 - INFO - __main__ - [Dataset] num_image_token: 256
07/03/2026 09:55:18 - INFO - __main__ - [Dataset] dynamic_image_size: True
07/03/2026 09:55:18 - INFO - __main__ - [Dataset] use_thumbnail: True
07/03/2026 09:55:18 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 12
07/03/2026 09:55:18 - INFO - __main__ - Formatting inputs...Skip in lazy mode
07/03/2026 09:55:19 - INFO - __main__ - Add dataset: vi-envelope-vqa with length: 8843
trainable params: 8,798,208 || all params: 638,496,128 || trainable%: 1.3779579255334184
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:19 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight
07/03/2026 09:55:20 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight
07/03/2026 09:55:20 - INFO - __main__ - mlp1.0.weight
07/03/2026 09:55:20 - INFO - __main__ - mlp1.0.bias
07/03/2026 09:55:20 - INFO - __main__ - mlp1.1.weight
07/03/2026 09:55:20 - INFO - __main__ - mlp1.1.bias
07/03/2026 09:55:20 - INFO - __main__ - mlp1.3.weight
07/03/2026 09:55:20 - INFO - __main__ - mlp1.3.bias
[INFO|trainer.py:641] 2026-07-03 09:55:20,040 >> Using auto half precision backend
[INFO|trainer.py:2078] 2026-07-03 09:55:21,739 >> ***** Running training *****
[INFO|trainer.py:2079] 2026-07-03 09:55:21,739 >>   Num examples = 8,843
[INFO|trainer.py:2080] 2026-07-03 09:55:21,740 >>   Num Epochs = 1
[INFO|trainer.py:2081] 2026-07-03 09:55:21,740 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2084] 2026-07-03 09:55:21,740 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:2085] 2026-07-03 09:55:21,740 >>   Gradient Accumulation steps = 128
[INFO|trainer.py:2086] 2026-07-03 09:55:21,740 >>   Total optimization steps = 69
[INFO|trainer.py:2087] 2026-07-03 09:55:21,747 >>   Number of trainable parameters = 13,281,024
[INFO|integration_utils.py:723] 2026-07-03 09:55:21,755 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
wandb: Currently logged in as: hhung2662000 (hhung26062000) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run nun5q9od
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in /root/ai-project/Vintern/internvl_chat/wandb/run-20260703_095522-nun5q9od
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run vintern_lora_card
wandb: ⭐️ View project at https://wandb.ai/hhung26062000/huggingface
wandb: 🚀 View run at https://wandb.ai/hhung26062000/huggingface/runs/nun5q9od

  0%|          | 0/69 [00:00<?, ?it/s]/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.09 | optimizer_gradients: 13.19 | optimizer_step: 6.09
[Rank 0] time (ms) | fwd_microstep: 191177.54 | bwd_microstep: 200125.18 | bwd_inner_microstep: 196746.03 | bwd_allreduce_microstep: 1320.49 | step_microstep: 68.36
[Rank 0] time (ms) | fwd: 191179.20 | bwd: 200068.85 | bwd_inner: 197058.00 | bwd_allreduce: 1337.85 | step: 68.36

  1%|▏         | 1/69 [08:35<9:43:58, 515.27s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.85 | optimizer_step: 0.94
[Rank 0] time (ms) | fwd_microstep: 191505.53 | bwd_microstep: 198396.96 | bwd_inner_microstep: 195179.22 | bwd_allreduce_microstep: 1092.03 | step_microstep: 25.24
[Rank 0] time (ms) | fwd: 191491.90 | bwd: 198361.62 | bwd_inner: 195574.16 | bwd_allreduce: 1071.52 | step: 25.23

  3%|▎         | 2/69 [16:22<9:03:44, 486.94s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.96
[Rank 0] time (ms) | fwd_microstep: 192959.71 | bwd_microstep: 205960.19 | bwd_inner_microstep: 202807.99 | bwd_allreduce_microstep: 1150.49 | step_microstep: 19.38
[Rank 0] time (ms) | fwd: 192944.56 | bwd: 205950.55 | bwd_inner: 203089.07 | bwd_allreduce: 1146.82 | step: 19.38

  4%|▍         | 3/69 [24:15<8:48:30, 480.47s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 193226.04 | bwd_microstep: 206922.58 | bwd_inner_microstep: 203528.37 | bwd_allreduce_microstep: 1262.27 | step_microstep: 20.75
[Rank 0] time (ms) | fwd: 193212.54 | bwd: 207049.23 | bwd_inner: 203858.87 | bwd_allreduce: 1276.72 | step: 20.75

  6%|▌         | 4/69 [32:10<8:38:18, 478.45s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.18 | optimizer_gradients: 4.38 | optimizer_step: 1.57
[Rank 0] time (ms) | fwd_microstep: 195001.94 | bwd_microstep: 209211.71 | bwd_inner_microstep: 205369.31 | bwd_allreduce_microstep: 1200.17 | step_microstep: 68.80
[Rank 0] time (ms) | fwd: 194996.29 | bwd: 209140.00 | bwd_inner: 205624.51 | bwd_allreduce: 1384.13 | step: 68.80

  7%|▋         | 5/69 [40:15<8:32:42, 480.67s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.80 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 192936.40 | bwd_microstep: 202554.43 | bwd_inner_microstep: 198927.17 | bwd_allreduce_microstep: 1227.85 | step_microstep: 22.56
[Rank 0] time (ms) | fwd: 192929.57 | bwd: 202436.50 | bwd_inner: 199183.49 | bwd_allreduce: 1244.71 | step: 22.56

  9%|▊         | 6/69 [48:05<8:21:00, 477.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
[2026-07-03 10:46:39,662] [WARNING] [real_accelerator.py:199:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
Traceback (most recent call last):
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 847, in <module>
    main()
  File "/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py", line 608, in main
    init_dist(launcher=launcher, backend='nccl')
  File "/root/ai-project/Vintern/internvl_chat/internvl/dist_utils.py", line 36, in init_dist
    _init_dist_pytorch(backend, **kwargs)
  File "/root/ai-project/Vintern/internvl_chat/internvl/dist_utils.py", line 49, in _init_dist_pytorch
    torch.cuda.set_device(rank % num_gpus)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py", line 420, in set_device
    torch._C._cuda_setDevice(device)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/cuda/__init__.py", line 314, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination
E0703 10:46:49.798000 126501186553664 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 36893) of binary: /root/miniconda3/envs/train_vlm/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/train_vlm/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/ai-project/Vintern/internvl_chat/internvl/train/internvl_chat_finetune.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-07-03_10:46:49
  host      : data-ai-gpu-01
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 36893)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/03/2026 10:47:39 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/03/2026 10:47:39 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=256,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul03_10-47-39_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=10,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=1.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/03/2026 10:47:39 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-03 10:47:39,476 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 10:47:39,476 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-03 10:47:39,476 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 10:47:39,476 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 10:47:39,476 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 10:47:39,476 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-03 10:47:39,892 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/03/2026 10:47:39 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-03 10:47:39,906 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-03 10:47:39,909 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/03/2026 10:47:39 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-03 10:47:39,910 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-03 10:47:39,945 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-03 10:47:39,948 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-03 10:47:40,029 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-03 10:47:42,205 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-03 10:47:42,205 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-03 10:47:42,211 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-03 10:47:42,212 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/03/2026 10:47:42 - INFO - __main__ - Finished
07/03/2026 10:47:42 - INFO - __main__ - model.config.force_image_size: 448
07/03/2026 10:47:42 - INFO - __main__ - data_args.force_image_size: 448
07/03/2026 10:47:42 - INFO - __main__ - model.config.vision_config.image_size: 448
07/03/2026 10:47:42 - INFO - __main__ - max_dynamic_patch is set to 12 according to the meta file
07/03/2026 10:47:42 - INFO - __main__ - [Dataset] num_image_token: 256
07/03/2026 10:47:42 - INFO - __main__ - [Dataset] dynamic_image_size: True
07/03/2026 10:47:42 - INFO - __main__ - [Dataset] use_thumbnail: True
07/03/2026 10:47:42 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 12
07/03/2026 10:47:42 - INFO - __main__ - Formatting inputs...Skip in lazy mode
07/03/2026 10:47:42 - INFO - __main__ - Add dataset: vi-envelope-vqa with length: 8843
trainable params: 8,798,208 || all params: 638,496,128 || trainable%: 1.3779579255334184
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight
07/03/2026 10:47:43 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight
07/03/2026 10:47:43 - INFO - __main__ - mlp1.0.weight
07/03/2026 10:47:43 - INFO - __main__ - mlp1.0.bias
07/03/2026 10:47:43 - INFO - __main__ - mlp1.1.weight
07/03/2026 10:47:43 - INFO - __main__ - mlp1.1.bias
07/03/2026 10:47:43 - INFO - __main__ - mlp1.3.weight
07/03/2026 10:47:43 - INFO - __main__ - mlp1.3.bias
[INFO|trainer.py:641] 2026-07-03 10:47:43,294 >> Using auto half precision backend
[INFO|trainer.py:2078] 2026-07-03 10:47:45,224 >> ***** Running training *****
[INFO|trainer.py:2079] 2026-07-03 10:47:45,224 >>   Num examples = 8,843
[INFO|trainer.py:2080] 2026-07-03 10:47:45,224 >>   Num Epochs = 1
[INFO|trainer.py:2081] 2026-07-03 10:47:45,224 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2084] 2026-07-03 10:47:45,224 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:2085] 2026-07-03 10:47:45,224 >>   Gradient Accumulation steps = 256
[INFO|trainer.py:2086] 2026-07-03 10:47:45,224 >>   Total optimization steps = 34
[INFO|trainer.py:2087] 2026-07-03 10:47:45,229 >>   Number of trainable parameters = 13,281,024
[INFO|integration_utils.py:723] 2026-07-03 10:47:45,234 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
wandb: Currently logged in as: hhung2662000 (hhung26062000) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run eum6zsve
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in /root/ai-project/Vintern/internvl_chat/wandb/run-20260703_104745-eum6zsve
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run vintern_lora_card
wandb: ⭐️ View project at https://wandb.ai/hhung26062000/huggingface
wandb: 🚀 View run at https://wandb.ai/hhung26062000/huggingface/runs/eum6zsve

  0%|          | 0/34 [00:00<?, ?it/s]/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
[Rank 0] time (ms) | optimizer_allgather: 0.11 | optimizer_gradients: 13.20 | optimizer_step: 6.33
[Rank 0] time (ms) | fwd_microstep: 387594.05 | bwd_microstep: 403730.25 | bwd_inner_microstep: 396484.57 | bwd_allreduce_microstep: 2596.03 | step_microstep: 71.96
[Rank 0] time (ms) | fwd: 387576.34 | bwd: 403523.60 | bwd_inner: 397210.44 | bwd_allreduce: 2592.51 | step: 71.96

  3%|▎         | 1/34 [16:29<9:04:05, 989.25s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.49 | optimizer_step: 0.97
[Rank 0] time (ms) | fwd_microstep: 384182.31 | bwd_microstep: 409407.31 | bwd_inner_microstep: 402552.59 | bwd_allreduce_microstep: 2485.81 | step_microstep: 23.87
[Rank 0] time (ms) | fwd: 384163.04 | bwd: 409331.75 | bwd_inner: 403171.18 | bwd_allreduce: 2550.29 | step: 23.86

  6%|▌         | 2/34 [32:14<8:33:41, 963.16s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.24 | optimizer_step: 1.04
[Rank 0] time (ms) | fwd_microstep: 386346.58 | bwd_microstep: 413752.06 | bwd_inner_microstep: 406398.67 | bwd_allreduce_microstep: 2454.46 | step_microstep: 28.50
[Rank 0] time (ms) | fwd: 386328.27 | bwd: 413635.28 | bwd_inner: 407097.30 | bwd_allreduce: 2522.38 | step: 28.52

  9%|▉         | 3/34 [48:12<8:16:30, 960.97s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2022
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.51 | optimizer_step: 0.95
[Rank 0] time (ms) | fwd_microstep: 386857.45 | bwd_microstep: 410284.82 | bwd_inner_microstep: 403385.42 | bwd_allreduce_microstep: 2507.30 | step_microstep: 20.30
[Rank 0] time (ms) | fwd: 386838.50 | bwd: 410225.58 | bwd_inner: 404005.84 | bwd_allreduce: 2547.07 | step: 20.33

 12%|█▏        | 4/34 [1:04:26<8:03:02, 966.07s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 389445.14 | bwd_microstep: 409234.70 | bwd_inner_microstep: 402196.66 | bwd_allreduce_microstep: 2410.75 | step_microstep: 20.36
[Rank 0] time (ms) | fwd: 389437.45 | bwd: 409267.91 | bwd_inner: 402804.74 | bwd_allreduce: 2436.42 | step: 20.36

 15%|█▍        | 5/34 [1:20:22<7:45:07, 962.32s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3550
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.60 | optimizer_step: 1.12
[Rank 0] time (ms) | fwd_microstep: 391890.70 | bwd_microstep: 406408.12 | bwd_inner_microstep: 399807.58 | bwd_allreduce_microstep: 2458.17 | step_microstep: 24.66
[Rank 0] time (ms) | fwd: 391861.51 | bwd: 406216.52 | bwd_inner: 400231.69 | bwd_allreduce: 2474.07 | step: 24.68

 18%|█▊        | 6/34 [1:36:13<7:27:17, 958.49s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.74 | optimizer_step: 1.12
[Rank 0] time (ms) | fwd_microstep: 389726.60 | bwd_microstep: 410625.71 | bwd_inner_microstep: 403454.75 | bwd_allreduce_microstep: 2417.14 | step_microstep: 527.35
[Rank 0] time (ms) | fwd: 389702.72 | bwd: 410519.92 | bwd_inner: 404272.68 | bwd_allreduce: 2389.58 | step: 527.32

 21%|██        | 7/34 [1:52:09<7:10:59, 957.75s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 382884.71 | bwd_microstep: 408485.94 | bwd_inner_microstep: 401497.01 | bwd_allreduce_microstep: 2389.06 | step_microstep: 19.79
[Rank 0] time (ms) | fwd: 382868.42 | bwd: 408354.91 | bwd_inner: 402077.93 | bwd_allreduce: 2460.05 | step: 19.79

 24%|██▎       | 8/34 [2:07:58<6:53:48, 954.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.41 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 389090.81 | bwd_microstep: 408187.87 | bwd_inner_microstep: 401574.86 | bwd_allreduce_microstep: 2548.23 | step_microstep: 20.63
[Rank 0] time (ms) | fwd: 389065.10 | bwd: 408115.14 | bwd_inner: 402041.26 | bwd_allreduce: 2603.26 | step: 20.63

 26%|██▋       | 9/34 [2:23:45<6:36:55, 952.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 969
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3534
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.94
[Rank 0] time (ms) | fwd_microstep: 389594.26 | bwd_microstep: 404868.90 | bwd_inner_microstep: 397674.83 | bwd_allreduce_microstep: 2508.98 | step_microstep: 19.65
[Rank 0] time (ms) | fwd: 389581.64 | bwd: 404814.51 | bwd_inner: 398226.99 | bwd_allreduce: 2668.91 | step: 19.65

 29%|██▉       | 10/34 [2:39:39<6:21:09, 952.91s/it]
                                                    
{'loss': 0.1117, 'grad_norm': 0.0006719666998833418, 'learning_rate': 0.00017071067811865476, 'epoch': 0.29}

 29%|██▉       | 10/34 [2:39:39<6:21:09, 952.91s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3539
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.98
[Rank 0] time (ms) | fwd_microstep: 392789.06 | bwd_microstep: 415288.64 | bwd_inner_microstep: 408848.98 | bwd_allreduce_microstep: 2226.95 | step_microstep: 18.94
[Rank 0] time (ms) | fwd: 392772.66 | bwd: 415121.34 | bwd_inner: 409419.84 | bwd_allreduce: 2176.14 | step: 18.94

 32%|███▏      | 11/34 [2:55:38<6:05:59, 954.74s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1983
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 386941.92 | bwd_microstep: 409139.78 | bwd_inner_microstep: 402004.56 | bwd_allreduce_microstep: 2675.94 | step_microstep: 19.56
[Rank 0] time (ms) | fwd: 386921.26 | bwd: 409103.01 | bwd_inner: 402567.76 | bwd_allreduce: 2659.81 | step: 19.57

 35%|███▌      | 12/34 [3:11:31<5:49:55, 954.33s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 978
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1956
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.57 | optimizer_step: 1.01
[Rank 0] time (ms) | fwd_microstep: 388196.24 | bwd_microstep: 420192.63 | bwd_inner_microstep: 412680.28 | bwd_allreduce_microstep: 2515.86 | step_microstep: 30.58
[Rank 0] time (ms) | fwd: 388171.42 | bwd: 420084.11 | bwd_inner: 413399.34 | bwd_allreduce: 2517.79 | step: 30.57

 38%|███▊      | 13/34 [3:27:31<5:34:34, 955.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 972
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 975
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/03/2026 14:29:08 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/03/2026 14:29:08 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=32,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul03_14-29-08_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=5,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=1.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=2,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/03/2026 14:29:08 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-03 14:29:08,758 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 14:29:08,758 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-03 14:29:08,758 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 14:29:08,758 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 14:29:08,758 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-03 14:29:08,759 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-03 14:29:09,554 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/03/2026 14:29:09 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-03 14:29:09,564 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-03 14:29:09,566 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/03/2026 14:29:09 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-03 14:29:09,567 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-03 14:29:09,591 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-03 14:29:09,592 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-03 14:29:09,651 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-03 14:29:11,568 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-03 14:29:11,569 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-03 14:29:11,585 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-03 14:29:11,586 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/03/2026 14:29:12 - INFO - __main__ - Finished
07/03/2026 14:29:12 - INFO - __main__ - model.config.force_image_size: 448
07/03/2026 14:29:12 - INFO - __main__ - data_args.force_image_size: 448
07/03/2026 14:29:12 - INFO - __main__ - model.config.vision_config.image_size: 448
07/03/2026 14:29:12 - INFO - __main__ - max_dynamic_patch is set to 12 according to the meta file
07/03/2026 14:29:12 - INFO - __main__ - [Dataset] num_image_token: 256
07/03/2026 14:29:12 - INFO - __main__ - [Dataset] dynamic_image_size: True
07/03/2026 14:29:12 - INFO - __main__ - [Dataset] use_thumbnail: True
07/03/2026 14:29:12 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 12
07/03/2026 14:29:12 - INFO - __main__ - Formatting inputs...Skip in lazy mode
07/03/2026 14:29:12 - INFO - __main__ - Add dataset: vi-envelope-vqa with length: 8843
trainable params: 8,798,208 || all params: 638,496,128 || trainable%: 1.3779579255334184
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight
07/03/2026 14:29:13 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight
07/03/2026 14:29:13 - INFO - __main__ - mlp1.0.weight
07/03/2026 14:29:13 - INFO - __main__ - mlp1.0.bias
07/03/2026 14:29:13 - INFO - __main__ - mlp1.1.weight
07/03/2026 14:29:13 - INFO - __main__ - mlp1.1.bias
07/03/2026 14:29:13 - INFO - __main__ - mlp1.3.weight
07/03/2026 14:29:13 - INFO - __main__ - mlp1.3.bias
[INFO|trainer.py:641] 2026-07-03 14:29:13,693 >> Using auto half precision backend
[INFO|trainer.py:2078] 2026-07-03 14:29:15,688 >> ***** Running training *****
[INFO|trainer.py:2079] 2026-07-03 14:29:15,688 >>   Num examples = 8,843
[INFO|trainer.py:2080] 2026-07-03 14:29:15,688 >>   Num Epochs = 1
[INFO|trainer.py:2081] 2026-07-03 14:29:15,689 >>   Instantaneous batch size per device = 2
[INFO|trainer.py:2084] 2026-07-03 14:29:15,689 >>   Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:2085] 2026-07-03 14:29:15,689 >>   Gradient Accumulation steps = 32
[INFO|trainer.py:2086] 2026-07-03 14:29:15,689 >>   Total optimization steps = 138
[INFO|trainer.py:2087] 2026-07-03 14:29:15,705 >>   Number of trainable parameters = 13,281,024
[INFO|integration_utils.py:723] 2026-07-03 14:29:15,722 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
wandb: Currently logged in as: hhung2662000 (hhung26062000) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run c0vco66y
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in /root/ai-project/Vintern/internvl_chat/wandb/run-20260703_142916-c0vco66y
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run vintern_lora_card
wandb: ⭐️ View project at https://wandb.ai/hhung26062000/huggingface
wandb: 🚀 View run at https://wandb.ai/hhung26062000/huggingface/runs/c0vco66y

  0%|          | 0/138 [00:00<?, ?it/s]/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.05 | optimizer_gradients: 10.55 | optimizer_step: 5.06
[Rank 0] time (ms) | fwd_microstep: 80001.50 | bwd_microstep: 84897.84 | bwd_inner_microstep: 84182.36 | bwd_allreduce_microstep: 344.60 | step_microstep: 56.32
[Rank 0] time (ms) | fwd: 80007.56 | bwd: 84945.86 | bwd_inner: 84317.15 | bwd_allreduce: 332.13 | step: 56.30

  1%|          | 1/138 [03:57<9:03:11, 237.89s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 78548.42 | bwd_microstep: 88740.95 | bwd_inner_microstep: 87303.41 | bwd_allreduce_microstep: 870.15 | step_microstep: 20.79
[Rank 0] time (ms) | fwd: 78553.44 | bwd: 88827.85 | bwd_inner: 87450.08 | bwd_allreduce: 882.63 | step: 20.80

  1%|▏         | 2/138 [07:07<7:54:57, 209.54s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.41 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 78486.62 | bwd_microstep: 82680.72 | bwd_inner_microstep: 81620.59 | bwd_allreduce_microstep: 678.67 | step_microstep: 25.93
[Rank 0] time (ms) | fwd: 78493.52 | bwd: 82734.44 | bwd_inner: 81743.03 | bwd_allreduce: 694.30 | step: 25.92

  2%|▏         | 3/138 [10:11<7:24:40, 197.63s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.65 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 78081.61 | bwd_microstep: 83457.13 | bwd_inner_microstep: 82624.94 | bwd_allreduce_microstep: 446.28 | step_microstep: 26.02
[Rank 0] time (ms) | fwd: 78090.08 | bwd: 83520.08 | bwd_inner: 82769.34 | bwd_allreduce: 432.03 | step: 26.02

  3%|▎         | 4/138 [13:13<7:08:00, 191.64s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.47 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 80654.28 | bwd_microstep: 85091.83 | bwd_inner_microstep: 84263.60 | bwd_allreduce_microstep: 395.46 | step_microstep: 25.72
[Rank 0] time (ms) | fwd: 80679.19 | bwd: 85164.91 | bwd_inner: 84393.20 | bwd_allreduce: 410.38 | step: 25.73

  4%|▎         | 5/138 [16:18<6:59:08, 189.09s/it]
                                                  
{'loss': 0.2047, 'grad_norm': 0.0248116422444582, 'learning_rate': 0.0002, 'epoch': 0.04}

  4%|▎         | 5/138 [16:18<6:59:08, 189.09s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.48 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 80477.83 | bwd_microstep: 86942.32 | bwd_inner_microstep: 85922.46 | bwd_allreduce_microstep: 575.79 | step_microstep: 20.67
[Rank 0] time (ms) | fwd: 80480.72 | bwd: 86994.97 | bwd_inner: 86086.49 | bwd_allreduce: 585.32 | step: 20.67

  4%|▍         | 6/138 [19:27<6:56:15, 189.21s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 80210.82 | bwd_microstep: 88565.55 | bwd_inner_microstep: 87504.26 | bwd_allreduce_microstep: 538.93 | step_microstep: 19.10
[Rank 0] time (ms) | fwd: 80219.10 | bwd: 88637.80 | bwd_inner: 87705.53 | bwd_allreduce: 540.17 | step: 19.11

  5%|▌         | 7/138 [22:36<6:53:09, 189.23s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 78513.42 | bwd_microstep: 84688.07 | bwd_inner_microstep: 83504.45 | bwd_allreduce_microstep: 679.18 | step_microstep: 20.92
[Rank 0] time (ms) | fwd: 78534.05 | bwd: 84761.54 | bwd_inner: 83673.87 | bwd_allreduce: 693.96 | step: 20.92

  6%|▌         | 8/138 [25:41<6:46:32, 187.64s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 78284.86 | bwd_microstep: 84071.58 | bwd_inner_microstep: 83117.34 | bwd_allreduce_microstep: 527.79 | step_microstep: 21.04
[Rank 0] time (ms) | fwd: 78292.58 | bwd: 84136.74 | bwd_inner: 83258.45 | bwd_allreduce: 520.98 | step: 21.04

  7%|▋         | 9/138 [28:44<6:40:51, 186.45s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 79678.53 | bwd_microstep: 83864.84 | bwd_inner_microstep: 82850.99 | bwd_allreduce_microstep: 514.87 | step_microstep: 19.59
[Rank 0] time (ms) | fwd: 79681.94 | bwd: 83933.16 | bwd_inner: 83017.49 | bwd_allreduce: 517.63 | step: 19.59

  7%|▋         | 10/138 [31:48<6:36:06, 185.68s/it]
                                                   
{'loss': 0.0396, 'grad_norm': 0.010104305110871792, 'learning_rate': 0.00019930337092856243, 'epoch': 0.07}

  7%|▋         | 10/138 [31:48<6:36:06, 185.68s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3533
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 81465.53 | bwd_microstep: 91669.64 | bwd_inner_microstep: 90713.59 | bwd_allreduce_microstep: 395.84 | step_microstep: 20.28
[Rank 0] time (ms) | fwd: 81484.79 | bwd: 91744.05 | bwd_inner: 90890.55 | bwd_allreduce: 380.42 | step: 20.29

  8%|▊         | 11/138 [35:03<6:38:36, 188.32s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.35 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 78419.34 | bwd_microstep: 85150.82 | bwd_inner_microstep: 83955.72 | bwd_allreduce_microstep: 676.16 | step_microstep: 20.75
[Rank 0] time (ms) | fwd: 78429.14 | bwd: 85211.23 | bwd_inner: 84126.97 | bwd_allreduce: 685.18 | step: 20.75

  9%|▊         | 12/138 [38:07<6:32:52, 187.09s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.05 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 78827.53 | bwd_microstep: 85630.04 | bwd_inner_microstep: 84812.73 | bwd_allreduce_microstep: 402.37 | step_microstep: 525.77
[Rank 0] time (ms) | fwd: 78835.74 | bwd: 85689.22 | bwd_inner: 84942.40 | bwd_allreduce: 367.79 | step: 525.76

  9%|▉         | 13/138 [41:14<6:29:56, 187.17s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2022
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.35 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 80287.78 | bwd_microstep: 84621.32 | bwd_inner_microstep: 83655.08 | bwd_allreduce_microstep: 534.36 | step_microstep: 19.79
[Rank 0] time (ms) | fwd: 80290.78 | bwd: 84700.72 | bwd_inner: 83792.47 | bwd_allreduce: 554.84 | step: 19.79

 10%|█         | 14/138 [44:20<6:26:07, 186.83s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 80887.03 | bwd_microstep: 83189.68 | bwd_inner_microstep: 82466.54 | bwd_allreduce_microstep: 329.60 | step_microstep: 22.37
[Rank 0] time (ms) | fwd: 80890.29 | bwd: 83244.32 | bwd_inner: 82612.06 | bwd_allreduce: 333.84 | step: 22.38

 11%|█         | 15/138 [47:25<6:21:42, 186.20s/it]
                                                   
{'loss': 0.027, 'grad_norm': 0.007679518777877092, 'learning_rate': 0.00019722318955551306, 'epoch': 0.11}

 11%|█         | 15/138 [47:25<6:21:42, 186.20s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.41 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 80931.80 | bwd_microstep: 89365.47 | bwd_inner_microstep: 88238.89 | bwd_allreduce_microstep: 585.41 | step_microstep: 19.33
[Rank 0] time (ms) | fwd: 80941.51 | bwd: 89431.35 | bwd_inner: 88402.02 | bwd_allreduce: 582.79 | step: 19.34

 12%|█▏        | 16/138 [50:35<6:21:04, 187.42s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3535
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 81033.70 | bwd_microstep: 89250.63 | bwd_inner_microstep: 88191.49 | bwd_allreduce_microstep: 537.57 | step_microstep: 20.25
[Rank 0] time (ms) | fwd: 81035.67 | bwd: 89325.82 | bwd_inner: 88353.98 | bwd_allreduce: 563.11 | step: 20.26

 12%|█▏        | 17/138 [53:45<6:19:18, 188.08s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.28 | optimizer_step: 1.00
[Rank 0] time (ms) | fwd_microstep: 80928.00 | bwd_microstep: 87739.21 | bwd_inner_microstep: 86693.70 | bwd_allreduce_microstep: 552.09 | step_microstep: 35.27
[Rank 0] time (ms) | fwd: 80953.37 | bwd: 87823.49 | bwd_inner: 86844.90 | bwd_allreduce: 563.19 | step: 35.27

 13%|█▎        | 18/138 [56:54<6:16:59, 188.49s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.55 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 79702.43 | bwd_microstep: 86953.67 | bwd_inner_microstep: 85878.81 | bwd_allreduce_microstep: 607.83 | step_microstep: 22.38
[Rank 0] time (ms) | fwd: 79711.41 | bwd: 87027.19 | bwd_inner: 86049.50 | bwd_allreduce: 527.68 | step: 22.38

 14%|█▍        | 19/138 [1:00:01<6:12:35, 187.87s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3532
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 79940.23 | bwd_microstep: 87414.83 | bwd_inner_microstep: 86340.97 | bwd_allreduce_microstep: 568.30 | step_microstep: 19.28
[Rank 0] time (ms) | fwd: 79962.38 | bwd: 87521.22 | bwd_inner: 86508.24 | bwd_allreduce: 582.45 | step: 19.28

 14%|█▍        | 20/138 [1:03:10<6:10:27, 188.37s/it]
                                                     
{'loss': 0.0175, 'grad_norm': 0.003990677185356617, 'learning_rate': 0.00019378843817721854, 'epoch': 0.14}

 14%|█▍        | 20/138 [1:03:10<6:10:27, 188.37s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3550
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 79858.08 | bwd_microstep: 83681.58 | bwd_inner_microstep: 82766.27 | bwd_allreduce_microstep: 501.27 | step_microstep: 19.32
[Rank 0] time (ms) | fwd: 79866.13 | bwd: 83761.44 | bwd_inner: 82879.14 | bwd_allreduce: 502.75 | step: 19.34

 15%|█▌        | 21/138 [1:06:16<6:05:34, 187.47s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.35 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 80465.52 | bwd_microstep: 86310.78 | bwd_inner_microstep: 85511.81 | bwd_allreduce_microstep: 429.35 | step_microstep: 20.24
[Rank 0] time (ms) | fwd: 80469.42 | bwd: 86373.19 | bwd_inner: 85625.09 | bwd_allreduce: 425.18 | step: 20.24

 16%|█▌        | 22/138 [1:09:23<6:02:33, 187.53s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3545
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.47 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 85917.96 | bwd_microstep: 89963.17 | bwd_inner_microstep: 88848.98 | bwd_allreduce_microstep: 533.88 | step_microstep: 20.21
[Rank 0] time (ms) | fwd: 85927.99 | bwd: 90038.43 | bwd_inner: 88992.09 | bwd_allreduce: 539.21 | step: 20.21

 17%|█▋        | 23/138 [1:12:38<6:03:16, 189.54s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.41 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 78830.83 | bwd_microstep: 83820.30 | bwd_inner_microstep: 82886.46 | bwd_allreduce_microstep: 464.83 | step_microstep: 21.13
[Rank 0] time (ms) | fwd: 78843.98 | bwd: 83885.09 | bwd_inner: 83058.62 | bwd_allreduce: 438.95 | step: 21.14

 17%|█▋        | 24/138 [1:15:40<5:56:14, 187.50s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.49 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 78622.11 | bwd_microstep: 86021.42 | bwd_inner_microstep: 85124.99 | bwd_allreduce_microstep: 516.89 | step_microstep: 29.31
[Rank 0] time (ms) | fwd: 78621.27 | bwd: 86093.11 | bwd_inner: 85246.76 | bwd_allreduce: 512.52 | step: 29.29

 18%|█▊        | 25/138 [1:18:46<5:51:53, 186.84s/it]
                                                     
{'loss': 0.017, 'grad_norm': 0.006311236415058374, 'learning_rate': 0.00018904697174694447, 'epoch': 0.18}

 18%|█▊        | 25/138 [1:18:46<5:51:53, 186.84s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.80 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 77975.01 | bwd_microstep: 83307.06 | bwd_inner_microstep: 82248.31 | bwd_allreduce_microstep: 592.92 | step_microstep: 21.94
[Rank 0] time (ms) | fwd: 77985.69 | bwd: 83392.14 | bwd_inner: 82394.83 | bwd_allreduce: 595.22 | step: 21.95

 19%|█▉        | 26/138 [1:21:48<5:46:28, 185.61s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.22 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 80199.12 | bwd_microstep: 83841.71 | bwd_inner_microstep: 82965.96 | bwd_allreduce_microstep: 466.01 | step_microstep: 34.71
[Rank 0] time (ms) | fwd: 80208.19 | bwd: 83923.23 | bwd_inner: 83092.92 | bwd_allreduce: 471.89 | step: 34.72

 20%|█▉        | 27/138 [1:24:53<5:42:46, 185.28s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 79359.73 | bwd_microstep: 85846.80 | bwd_inner_microstep: 84972.69 | bwd_allreduce_microstep: 390.47 | step_microstep: 20.09
[Rank 0] time (ms) | fwd: 79370.64 | bwd: 85917.88 | bwd_inner: 85133.95 | bwd_allreduce: 410.40 | step: 20.10

 20%|██        | 28/138 [1:27:58<5:39:46, 185.33s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3532
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.55 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 82413.70 | bwd_microstep: 87343.24 | bwd_inner_microstep: 86501.12 | bwd_allreduce_microstep: 451.22 | step_microstep: 21.64
[Rank 0] time (ms) | fwd: 82429.22 | bwd: 87408.41 | bwd_inner: 86623.93 | bwd_allreduce: 461.92 | step: 21.64

 21%|██        | 29/138 [1:31:08<5:39:13, 186.73s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.69 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 80480.19 | bwd_microstep: 85830.02 | bwd_inner_microstep: 84845.64 | bwd_allreduce_microstep: 532.22 | step_microstep: 22.04
[Rank 0] time (ms) | fwd: 80503.68 | bwd: 85920.47 | bwd_inner: 84995.13 | bwd_allreduce: 530.67 | step: 22.04

 22%|██▏       | 30/138 [1:34:15<5:35:49, 186.57s/it]
                                                     
{'loss': 0.0131, 'grad_norm': 0.004349194932729006, 'learning_rate': 0.0001830648511318223, 'epoch': 0.22}

 22%|██▏       | 30/138 [1:34:15<5:35:49, 186.57s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 79324.79 | bwd_microstep: 84350.48 | bwd_inner_microstep: 83405.98 | bwd_allreduce_microstep: 510.33 | step_microstep: 21.39
[Rank 0] time (ms) | fwd: 79333.75 | bwd: 84418.41 | bwd_inner: 83571.03 | bwd_allreduce: 507.30 | step: 21.40

 22%|██▏       | 31/138 [1:37:20<5:31:54, 186.12s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 77873.57 | bwd_microstep: 86264.45 | bwd_inner_microstep: 85023.58 | bwd_allreduce_microstep: 715.03 | step_microstep: 20.59
[Rank 0] time (ms) | fwd: 77885.59 | bwd: 86329.33 | bwd_inner: 85172.29 | bwd_allreduce: 699.87 | step: 20.60

 23%|██▎       | 32/138 [1:40:25<5:28:21, 185.86s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.18 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 78680.65 | bwd_microstep: 83833.93 | bwd_inner_microstep: 82880.43 | bwd_allreduce_microstep: 519.33 | step_microstep: 29.77
[Rank 0] time (ms) | fwd: 78688.03 | bwd: 83896.18 | bwd_inner: 83028.47 | bwd_allreduce: 511.86 | step: 29.76

 24%|██▍       | 33/138 [1:43:28<5:23:55, 185.10s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 79121.86 | bwd_microstep: 83135.50 | bwd_inner_microstep: 82268.92 | bwd_allreduce_microstep: 489.42 | step_microstep: 19.60
[Rank 0] time (ms) | fwd: 79135.23 | bwd: 83196.21 | bwd_inner: 82403.27 | bwd_allreduce: 497.97 | step: 19.61

 25%|██▍       | 34/138 [1:46:33<5:20:32, 184.93s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 78501.42 | bwd_microstep: 84704.32 | bwd_inner_microstep: 83645.88 | bwd_allreduce_microstep: 593.57 | step_microstep: 19.42
[Rank 0] time (ms) | fwd: 78511.29 | bwd: 84769.68 | bwd_inner: 83803.33 | bwd_allreduce: 559.12 | step: 19.42

 25%|██▌       | 35/138 [1:49:37<5:17:01, 184.67s/it]
                                                     
{'loss': 0.0124, 'grad_norm': 0.004049016162753105, 'learning_rate': 0.00017592542271443887, 'epoch': 0.25}

 25%|██▌       | 35/138 [1:49:37<5:17:01, 184.67s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3545
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.41 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 81632.57 | bwd_microstep: 88384.32 | bwd_inner_microstep: 87244.92 | bwd_allreduce_microstep: 595.64 | step_microstep: 20.95
[Rank 0] time (ms) | fwd: 81634.98 | bwd: 88463.44 | bwd_inner: 87423.41 | bwd_allreduce: 583.80 | step: 20.95

 26%|██▌       | 36/138 [1:52:47<5:16:51, 186.38s/it]dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3532
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
[Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 1.39 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 84774.79 | bwd_microstep: 90988.38 | bwd_inner_microstep: 89764.04 | bwd_allreduce_microstep: 820.07 | step_microstep: 21.15
[Rank 0] time (ms) | fwd: 84783.63 | bwd: 91068.76 | bwd_inner: 89908.40 | bwd_allreduce: 820.91 | step: 21.12

 27%|██▋       | 37/138 [1:56:01<5:17:40, 188.72s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 80771.92 | bwd_microstep: 88490.28 | bwd_inner_microstep: 87261.60 | bwd_allreduce_microstep: 669.14 | step_microstep: 21.44
[Rank 0] time (ms) | fwd: 80778.44 | bwd: 88611.64 | bwd_inner: 87441.03 | bwd_allreduce: 659.80 | step: 21.44

 28%|██▊       | 38/138 [1:59:12<5:15:29, 189.29s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.52 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 83932.69 | bwd_microstep: 88515.50 | bwd_inner_microstep: 87430.31 | bwd_allreduce_microstep: 636.97 | step_microstep: 20.75
[Rank 0] time (ms) | fwd: 83952.16 | bwd: 88584.54 | bwd_inner: 87574.97 | bwd_allreduce: 613.65 | step: 20.78

 28%|██▊       | 39/138 [2:02:24<5:13:28, 189.99s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3534
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 82101.96 | bwd_microstep: 87333.12 | bwd_inner_microstep: 86388.80 | bwd_allreduce_microstep: 486.92 | step_microstep: 21.68
[Rank 0] time (ms) | fwd: 82141.44 | bwd: 87400.27 | bwd_inner: 86528.65 | bwd_allreduce: 495.47 | step: 21.66

 29%|██▉       | 40/138 [2:05:33<5:10:03, 189.83s/it]
                                                     
{'loss': 0.012, 'grad_norm': 0.0037118764594197273, 'learning_rate': 0.00016772815716257412, 'epoch': 0.29}

 29%|██▉       | 40/138 [2:05:33<5:10:03, 189.83s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.35 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 78177.73 | bwd_microstep: 85683.67 | bwd_inner_microstep: 84477.25 | bwd_allreduce_microstep: 616.08 | step_microstep: 21.98
[Rank 0] time (ms) | fwd: 78180.60 | bwd: 85766.20 | bwd_inner: 84663.13 | bwd_allreduce: 619.27 | step: 22.00

 30%|██▉       | 41/138 [2:08:36<5:03:38, 187.82s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.61 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 79570.12 | bwd_microstep: 86713.99 | bwd_inner_microstep: 85782.93 | bwd_allreduce_microstep: 436.18 | step_microstep: 23.96
[Rank 0] time (ms) | fwd: 79577.95 | bwd: 86790.08 | bwd_inner: 85955.31 | bwd_allreduce: 438.21 | step: 23.96

 30%|███       | 42/138 [2:11:44<5:00:18, 187.69s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3539
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3537
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 82322.48 | bwd_microstep: 89493.47 | bwd_inner_microstep: 88469.86 | bwd_allreduce_microstep: 569.80 | step_microstep: 19.94
[Rank 0] time (ms) | fwd: 82334.07 | bwd: 89554.48 | bwd_inner: 88627.78 | bwd_allreduce: 557.50 | step: 19.94

 31%|███       | 43/138 [2:14:56<4:59:18, 189.04s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 79301.05 | bwd_microstep: 85492.83 | bwd_inner_microstep: 84399.29 | bwd_allreduce_microstep: 645.99 | step_microstep: 20.06
[Rank 0] time (ms) | fwd: 79316.62 | bwd: 85559.00 | bwd_inner: 84540.76 | bwd_allreduce: 663.67 | step: 20.07

 32%|███▏      | 44/138 [2:18:02<4:54:58, 188.28s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 78590.23 | bwd_microstep: 83530.62 | bwd_inner_microstep: 82581.45 | bwd_allreduce_microstep: 495.94 | step_microstep: 27.85
[Rank 0] time (ms) | fwd: 78596.63 | bwd: 83593.59 | bwd_inner: 82773.76 | bwd_allreduce: 495.93 | step: 27.86

 33%|███▎      | 45/138 [2:21:05<4:49:17, 186.64s/it]
                                                     
{'loss': 0.0138, 'grad_norm': 0.0033980372827500105, 'learning_rate': 0.00015858726354602248, 'epoch': 0.33}

 33%|███▎      | 45/138 [2:21:05<4:49:17, 186.64s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3532
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.66 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 81385.41 | bwd_microstep: 85894.60 | bwd_inner_microstep: 84851.33 | bwd_allreduce_microstep: 529.55 | step_microstep: 26.44
[Rank 0] time (ms) | fwd: 81389.68 | bwd: 85965.85 | bwd_inner: 85022.71 | bwd_allreduce: 512.86 | step: 26.43

 33%|███▎      | 46/138 [2:24:14<4:47:25, 187.45s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 78001.05 | bwd_microstep: 83455.92 | bwd_inner_microstep: 82685.58 | bwd_allreduce_microstep: 373.24 | step_microstep: 22.36
[Rank 0] time (ms) | fwd: 78002.21 | bwd: 83508.33 | bwd_inner: 82829.82 | bwd_allreduce: 355.11 | step: 22.36

 34%|███▍      | 47/138 [2:27:18<4:42:37, 186.35s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3535
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.93 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 80336.83 | bwd_microstep: 87714.93 | bwd_inner_microstep: 86689.35 | bwd_allreduce_microstep: 493.67 | step_microstep: 21.13
[Rank 0] time (ms) | fwd: 80352.86 | bwd: 87793.67 | bwd_inner: 86848.22 | bwd_allreduce: 484.69 | step: 21.13

 35%|███▍      | 48/138 [2:30:27<4:40:46, 187.19s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 79353.18 | bwd_microstep: 84082.03 | bwd_inner_microstep: 83130.09 | bwd_allreduce_microstep: 504.62 | step_microstep: 19.43
[Rank 0] time (ms) | fwd: 79360.27 | bwd: 84151.40 | bwd_inner: 83272.85 | bwd_allreduce: 485.56 | step: 19.43

 36%|███▌      | 49/138 [2:33:31<4:36:14, 186.23s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.97 | optimizer_gradients: 165.86 | optimizer_step: 31.47
[Rank 0] time (ms) | fwd_microstep: 78558.70 | bwd_microstep: 82645.27 | bwd_inner_microstep: 81974.75 | bwd_allreduce_microstep: 278.63 | step_microstep: 513.29
[Rank 0] time (ms) | fwd: 78573.13 | bwd: 82708.64 | bwd_inner: 82097.76 | bwd_allreduce: 283.50 | step: 513.39

 36%|███▌      | 50/138 [2:36:33<4:31:13, 184.92s/it]
                                                     
{'loss': 0.0136, 'grad_norm': 0.00322695542126894, 'learning_rate': 0.00014863009810942815, 'epoch': 0.36}

 36%|███▌      | 50/138 [2:36:33<4:31:13, 184.92s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.55 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 78199.39 | bwd_microstep: 84925.41 | bwd_inner_microstep: 83713.40 | bwd_allreduce_microstep: 749.80 | step_microstep: 26.72
[Rank 0] time (ms) | fwd: 78223.67 | bwd: 84980.66 | bwd_inner: 83860.29 | bwd_allreduce: 753.66 | step: 26.72

 37%|███▋      | 51/138 [2:39:36<4:27:12, 184.29s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.58 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 77932.29 | bwd_microstep: 81662.15 | bwd_inner_microstep: 80741.52 | bwd_allreduce_microstep: 507.68 | step_microstep: 25.78
[Rank 0] time (ms) | fwd: 77942.97 | bwd: 81721.15 | bwd_inner: 80854.87 | bwd_allreduce: 520.96 | step: 25.76

 38%|███▊      | 52/138 [2:42:37<4:22:45, 183.32s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 78532.01 | bwd_microstep: 84617.63 | bwd_inner_microstep: 83769.69 | bwd_allreduce_microstep: 441.76 | step_microstep: 19.15
[Rank 0] time (ms) | fwd: 78562.49 | bwd: 84671.44 | bwd_inner: 83908.06 | bwd_allreduce: 431.28 | step: 19.15

 38%|███▊      | 53/138 [2:45:40<4:19:42, 183.33s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.40 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 78246.27 | bwd_microstep: 84716.45 | bwd_inner_microstep: 83715.13 | bwd_allreduce_microstep: 494.22 | step_microstep: 27.38
[Rank 0] time (ms) | fwd: 78245.52 | bwd: 84795.84 | bwd_inner: 83893.05 | bwd_allreduce: 490.83 | step: 27.37

 39%|███▉      | 54/138 [2:48:44<4:16:45, 183.40s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.47 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 77947.90 | bwd_microstep: 83764.02 | bwd_inner_microstep: 82902.46 | bwd_allreduce_microstep: 476.03 | step_microstep: 21.37
[Rank 0] time (ms) | fwd: 77947.71 | bwd: 83818.46 | bwd_inner: 83027.35 | bwd_allreduce: 473.02 | step: 21.38

 40%|███▉      | 55/138 [2:51:46<4:12:57, 182.86s/it]
                                                     
{'loss': 0.0104, 'grad_norm': 0.004632092081010342, 'learning_rate': 0.000137995389871036, 'epoch': 0.4}

 40%|███▉      | 55/138 [2:51:46<4:12:57, 182.86s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 80182.97 | bwd_microstep: 88210.84 | bwd_inner_microstep: 87202.39 | bwd_allreduce_microstep: 540.53 | step_microstep: 20.10
[Rank 0] time (ms) | fwd: 80196.19 | bwd: 88279.03 | bwd_inner: 87369.88 | bwd_allreduce: 533.09 | step: 20.11

 41%|████      | 56/138 [2:54:56<4:12:53, 185.05s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 78713.69 | bwd_microstep: 83701.33 | bwd_inner_microstep: 82456.43 | bwd_allreduce_microstep: 608.09 | step_microstep: 20.33
[Rank 0] time (ms) | fwd: 78718.34 | bwd: 83772.69 | bwd_inner: 82637.98 | bwd_allreduce: 601.12 | step: 20.34

 41%|████▏     | 57/138 [2:57:59<4:09:16, 184.65s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.31 | optimizer_step: 1.03
[Rank 0] time (ms) | fwd_microstep: 79107.32 | bwd_microstep: 84409.08 | bwd_inner_microstep: 83691.85 | bwd_allreduce_microstep: 326.24 | step_microstep: 401.61
[Rank 0] time (ms) | fwd: 79103.63 | bwd: 84475.74 | bwd_inner: 83829.57 | bwd_allreduce: 333.70 | step: 401.60

 42%|████▏     | 58/138 [3:01:04<4:06:11, 184.64s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3532
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 80640.68 | bwd_microstep: 88938.82 | bwd_inner_microstep: 88040.02 | bwd_allreduce_microstep: 412.77 | step_microstep: 19.05
[Rank 0] time (ms) | fwd: 80649.20 | bwd: 88999.33 | bwd_inner: 88200.31 | bwd_allreduce: 428.69 | step: 19.05

 43%|████▎     | 59/138 [3:04:14<4:05:19, 186.33s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 76978.22 | bwd_microstep: 84331.87 | bwd_inner_microstep: 83259.21 | bwd_allreduce_microstep: 590.40 | step_microstep: 19.82
[Rank 0] time (ms) | fwd: 76981.82 | bwd: 84407.00 | bwd_inner: 83432.77 | bwd_allreduce: 596.25 | step: 19.83

 43%|████▎     | 60/138 [3:07:18<4:01:12, 185.55s/it]
                                                     
{'loss': 0.0129, 'grad_norm': 0.0040463777258992195, 'learning_rate': 0.0001268313077693485, 'epoch': 0.43}

 43%|████▎     | 60/138 [3:07:18<4:01:12, 185.55s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 77212.76 | bwd_microstep: 83206.88 | bwd_inner_microstep: 81950.85 | bwd_allreduce_microstep: 765.76 | step_microstep: 19.05
[Rank 0] time (ms) | fwd: 77209.92 | bwd: 83284.11 | bwd_inner: 82125.75 | bwd_allreduce: 772.21 | step: 19.05

 44%|████▍     | 61/138 [3:10:20<3:56:42, 184.45s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.63 | optimizer_step: 0.99
[Rank 0] time (ms) | fwd_microstep: 78511.91 | bwd_microstep: 84886.33 | bwd_inner_microstep: 83376.72 | bwd_allreduce_microstep: 1055.49 | step_microstep: 33.03
[Rank 0] time (ms) | fwd: 78524.21 | bwd: 84953.49 | bwd_inner: 83525.33 | bwd_allreduce: 1081.16 | step: 33.00

 45%|████▍     | 62/138 [3:13:23<3:53:09, 184.08s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.15 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 78916.49 | bwd_microstep: 85255.31 | bwd_inner_microstep: 84246.14 | bwd_allreduce_microstep: 545.35 | step_microstep: 34.27
[Rank 0] time (ms) | fwd: 78929.44 | bwd: 85339.58 | bwd_inner: 84392.82 | bwd_allreduce: 551.83 | step: 34.28

 46%|████▌     | 63/138 [3:16:28<3:50:25, 184.34s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 78860.25 | bwd_microstep: 85447.77 | bwd_inner_microstep: 84398.08 | bwd_allreduce_microstep: 543.79 | step_microstep: 19.60
[Rank 0] time (ms) | fwd: 78863.13 | bwd: 85529.95 | bwd_inner: 84541.26 | bwd_allreduce: 566.95 | step: 19.60

 46%|████▋     | 64/138 [3:19:32<3:47:14, 184.25s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 78165.36 | bwd_microstep: 82048.26 | bwd_inner_microstep: 81142.70 | bwd_allreduce_microstep: 487.85 | step_microstep: 20.14
[Rank 0] time (ms) | fwd: 78160.02 | bwd: 82111.50 | bwd_inner: 81277.15 | bwd_allreduce: 472.43 | step: 20.16

 47%|████▋     | 65/138 [3:22:33<3:42:58, 183.27s/it]
                                                     
{'loss': 0.0124, 'grad_norm': 0.0027721289079636335, 'learning_rate': 0.0001152933962873246, 'epoch': 0.47}

 47%|████▋     | 65/138 [3:22:33<3:42:58, 183.27s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.60 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 78674.98 | bwd_microstep: 84717.10 | bwd_inner_microstep: 83822.77 | bwd_allreduce_microstep: 354.10 | step_microstep: 47.20
[Rank 0] time (ms) | fwd: 78679.47 | bwd: 84787.35 | bwd_inner: 84022.35 | bwd_allreduce: 366.83 | step: 47.22

 48%|████▊     | 66/138 [3:25:38<3:40:28, 183.74s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3536
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.61 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 82332.38 | bwd_microstep: 86309.81 | bwd_inner_microstep: 85415.11 | bwd_allreduce_microstep: 496.36 | step_microstep: 23.29
[Rank 0] time (ms) | fwd: 82331.47 | bwd: 86383.21 | bwd_inner: 85553.91 | bwd_allreduce: 502.49 | step: 23.30

 49%|████▊     | 67/138 [3:28:47<3:39:10, 185.22s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3537
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.66 | optimizer_step: 1.01
[Rank 0] time (ms) | fwd_microstep: 81171.67 | bwd_microstep: 88784.62 | bwd_inner_microstep: 87574.04 | bwd_allreduce_microstep: 804.77 | step_microstep: 35.59
[Rank 0] time (ms) | fwd: 81175.00 | bwd: 88850.67 | bwd_inner: 87698.50 | bwd_allreduce: 814.88 | step: 35.59

 49%|████▉     | 68/138 [3:31:57<3:37:43, 186.62s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 80461.96 | bwd_microstep: 87622.86 | bwd_inner_microstep: 86527.05 | bwd_allreduce_microstep: 576.42 | step_microstep: 19.66
[Rank 0] time (ms) | fwd: 80472.76 | bwd: 87710.40 | bwd_inner: 86702.76 | bwd_allreduce: 591.22 | step: 19.66

 50%|█████     | 69/138 [3:35:05<3:35:17, 187.21s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 79488.63 | bwd_microstep: 85695.07 | bwd_inner_microstep: 84670.74 | bwd_allreduce_microstep: 577.47 | step_microstep: 20.63
[Rank 0] time (ms) | fwd: 79503.83 | bwd: 85768.83 | bwd_inner: 84824.61 | bwd_allreduce: 573.55 | step: 20.63

 51%|█████     | 70/138 [3:38:09<3:30:58, 186.15s/it]
                                                     
{'loss': 0.0117, 'grad_norm': 0.0036459509283304214, 'learning_rate': 0.00010354240831620541, 'epoch': 0.51}

 51%|█████     | 70/138 [3:38:09<3:30:58, 186.15s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.47 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 77059.35 | bwd_microstep: 83385.04 | bwd_inner_microstep: 82128.07 | bwd_allreduce_microstep: 854.52 | step_microstep: 35.78
[Rank 0] time (ms) | fwd: 77063.53 | bwd: 83453.00 | bwd_inner: 82241.52 | bwd_allreduce: 859.94 | step: 35.78

 51%|█████▏    | 71/138 [3:41:10<3:26:14, 184.70s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 80151.34 | bwd_microstep: 86310.25 | bwd_inner_microstep: 85363.79 | bwd_allreduce_microstep: 476.58 | step_microstep: 19.93
[Rank 0] time (ms) | fwd: 80153.36 | bwd: 86378.46 | bwd_inner: 85532.41 | bwd_allreduce: 465.23 | step: 19.94

 52%|█████▏    | 72/138 [3:44:17<3:23:53, 185.36s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 77617.95 | bwd_microstep: 85761.79 | bwd_inner_microstep: 84680.49 | bwd_allreduce_microstep: 567.44 | step_microstep: 21.45
[Rank 0] time (ms) | fwd: 77627.43 | bwd: 85838.51 | bwd_inner: 84846.84 | bwd_allreduce: 556.30 | step: 21.45

 53%|█████▎    | 73/138 [3:47:20<3:20:04, 184.69s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3530
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 82221.57 | bwd_microstep: 88501.12 | bwd_inner_microstep: 87493.12 | bwd_allreduce_microstep: 512.65 | step_microstep: 18.77
[Rank 0] time (ms) | fwd: 82233.76 | bwd: 88569.37 | bwd_inner: 87683.70 | bwd_allreduce: 509.50 | step: 18.77

 54%|█████▎    | 74/138 [3:50:32<3:19:12, 186.75s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 79484.56 | bwd_microstep: 85341.88 | bwd_inner_microstep: 84340.93 | bwd_allreduce_microstep: 552.51 | step_microstep: 20.30
[Rank 0] time (ms) | fwd: 79494.19 | bwd: 85404.34 | bwd_inner: 84491.30 | bwd_allreduce: 557.96 | step: 20.28

 54%|█████▍    | 75/138 [3:53:38<3:15:49, 186.50s/it]
                                                     
{'loss': 0.0111, 'grad_norm': 0.005587196908891201, 'learning_rate': 9.174206545276677e-05, 'epoch': 0.54}

 54%|█████▍    | 75/138 [3:53:38<3:15:49, 186.50s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 79490.54 | bwd_microstep: 83987.75 | bwd_inner_microstep: 82860.62 | bwd_allreduce_microstep: 601.91 | step_microstep: 19.74
[Rank 0] time (ms) | fwd: 79492.74 | bwd: 84061.58 | bwd_inner: 83038.93 | bwd_allreduce: 620.66 | step: 19.74

 55%|█████▌    | 76/138 [3:56:43<3:12:14, 186.04s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.76 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 79756.12 | bwd_microstep: 84908.37 | bwd_inner_microstep: 83810.05 | bwd_allreduce_microstep: 534.41 | step_microstep: 546.69
[Rank 0] time (ms) | fwd: 79761.42 | bwd: 84997.05 | bwd_inner: 83989.51 | bwd_allreduce: 538.75 | step: 546.61

 56%|█████▌    | 77/138 [3:59:49<3:09:16, 186.17s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.48 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 79617.52 | bwd_microstep: 87804.49 | bwd_inner_microstep: 86776.31 | bwd_allreduce_microstep: 519.56 | step_microstep: 20.85
[Rank 0] time (ms) | fwd: 79627.69 | bwd: 87876.49 | bwd_inner: 86947.66 | bwd_allreduce: 506.27 | step: 20.85

 57%|█████▋    | 78/138 [4:02:57<3:06:43, 186.72s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 77013.33 | bwd_microstep: 82939.64 | bwd_inner_microstep: 82133.76 | bwd_allreduce_microstep: 373.94 | step_microstep: 18.55
[Rank 0] time (ms) | fwd: 77024.11 | bwd: 83044.36 | bwd_inner: 82273.76 | bwd_allreduce: 392.03 | step: 18.56

 57%|█████▋    | 79/138 [4:05:59<3:02:20, 185.43s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.61 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 79037.17 | bwd_microstep: 82398.51 | bwd_inner_microstep: 81434.91 | bwd_allreduce_microstep: 569.98 | step_microstep: 21.27
[Rank 0] time (ms) | fwd: 79046.50 | bwd: 82468.50 | bwd_inner: 81554.14 | bwd_allreduce: 553.88 | step: 21.28

 58%|█████▊    | 80/138 [4:09:01<2:57:59, 184.13s/it]
                                                     
{'loss': 0.0124, 'grad_norm': 0.004005922935903072, 'learning_rate': 8.005677693484077e-05, 'epoch': 0.58}

 58%|█████▊    | 80/138 [4:09:01<2:57:59, 184.13s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 79018.52 | bwd_microstep: 83802.21 | bwd_inner_microstep: 82933.04 | bwd_allreduce_microstep: 445.74 | step_microstep: 19.53
[Rank 0] time (ms) | fwd: 79038.51 | bwd: 83865.43 | bwd_inner: 83056.22 | bwd_allreduce: 423.69 | step: 19.54

 59%|█████▊    | 81/138 [4:12:04<2:54:43, 183.92s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 77855.38 | bwd_microstep: 83084.81 | bwd_inner_microstep: 82187.89 | bwd_allreduce_microstep: 488.68 | step_microstep: 20.12
[Rank 0] time (ms) | fwd: 77863.13 | bwd: 83154.27 | bwd_inner: 82314.95 | bwd_allreduce: 489.96 | step: 20.12

 59%|█████▉    | 82/138 [4:15:05<2:50:57, 183.17s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 12, images per sample: 6.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 77220.69 | bwd_microstep: 84452.19 | bwd_inner_microstep: 83403.86 | bwd_allreduce_microstep: 621.42 | step_microstep: 18.30
[Rank 0] time (ms) | fwd: 77222.36 | bwd: 84520.42 | bwd_inner: 83547.30 | bwd_allreduce: 598.98 | step: 18.30

 60%|██████    | 83/138 [4:18:08<2:47:37, 182.87s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.50 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 79003.03 | bwd_microstep: 87907.23 | bwd_inner_microstep: 86442.61 | bwd_allreduce_microstep: 893.09 | step_microstep: 27.14
[Rank 0] time (ms) | fwd: 79002.30 | bwd: 87964.16 | bwd_inner: 86777.25 | bwd_allreduce: 895.35 | step: 27.12

 61%|██████    | 84/138 [4:21:16<2:45:59, 184.43s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3535
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.59 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 81673.06 | bwd_microstep: 89768.48 | bwd_inner_microstep: 88717.37 | bwd_allreduce_microstep: 563.35 | step_microstep: 24.00
[Rank 0] time (ms) | fwd: 81676.02 | bwd: 89847.30 | bwd_inner: 88880.20 | bwd_allreduce: 529.04 | step: 24.00

 62%|██████▏   | 85/138 [4:24:27<2:44:46, 186.53s/it]
                                                     
{'loss': 0.0094, 'grad_norm': 0.002914112526923418, 'learning_rate': 6.864934899622191e-05, 'epoch': 0.62}

 62%|██████▏   | 85/138 [4:24:27<2:44:46, 186.53s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3530
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 79729.40 | bwd_microstep: 88090.07 | bwd_inner_microstep: 87031.09 | bwd_allreduce_microstep: 555.85 | step_microstep: 19.80
[Rank 0] time (ms) | fwd: 79743.28 | bwd: 88166.17 | bwd_inner: 87204.90 | bwd_allreduce: 557.26 | step: 19.80

 62%|██████▏   | 86/138 [4:27:34<2:41:49, 186.72s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.41 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 78767.97 | bwd_microstep: 82430.13 | bwd_inner_microstep: 81599.24 | bwd_allreduce_microstep: 444.60 | step_microstep: 19.75
[Rank 0] time (ms) | fwd: 78758.52 | bwd: 82507.79 | bwd_inner: 81737.61 | bwd_allreduce: 459.55 | step: 19.76

 63%|██████▎   | 87/138 [4:30:35<2:37:14, 184.98s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 77305.55 | bwd_microstep: 83029.00 | bwd_inner_microstep: 82043.77 | bwd_allreduce_microstep: 514.33 | step_microstep: 18.27
[Rank 0] time (ms) | fwd: 77314.24 | bwd: 83099.75 | bwd_inner: 82187.92 | bwd_allreduce: 529.06 | step: 18.27

 64%|██████▍   | 88/138 [4:33:36<2:33:05, 183.72s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.58 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 78597.66 | bwd_microstep: 82898.56 | bwd_inner_microstep: 82002.65 | bwd_allreduce_microstep: 501.24 | step_microstep: 23.26
[Rank 0] time (ms) | fwd: 78606.98 | bwd: 82955.94 | bwd_inner: 82122.06 | bwd_allreduce: 509.21 | step: 23.26

 64%|██████▍   | 89/138 [4:36:39<2:29:48, 183.44s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.58 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 80199.84 | bwd_microstep: 86226.23 | bwd_inner_microstep: 85333.46 | bwd_allreduce_microstep: 465.52 | step_microstep: 30.19
[Rank 0] time (ms) | fwd: 80204.01 | bwd: 86304.95 | bwd_inner: 85450.81 | bwd_allreduce: 480.55 | step: 30.18

 65%|██████▌   | 90/138 [4:39:45<2:27:29, 184.37s/it]
                                                     
{'loss': 0.0108, 'grad_norm': 0.0034878209698945284, 'learning_rate': 5.767871655555751e-05, 'epoch': 0.65}

 65%|██████▌   | 90/138 [4:39:45<2:27:29, 184.37s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 79201.48 | bwd_microstep: 83303.22 | bwd_inner_microstep: 82383.80 | bwd_allreduce_microstep: 465.46 | step_microstep: 18.79
[Rank 0] time (ms) | fwd: 79205.43 | bwd: 83364.65 | bwd_inner: 82552.13 | bwd_allreduce: 472.53 | step: 18.79

 66%|██████▌   | 91/138 [4:42:49<2:24:11, 184.07s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 78477.05 | bwd_microstep: 84547.64 | bwd_inner_microstep: 83663.45 | bwd_allreduce_microstep: 458.21 | step_microstep: 20.82
[Rank 0] time (ms) | fwd: 78480.84 | bwd: 84614.30 | bwd_inner: 83801.10 | bwd_allreduce: 454.54 | step: 20.83

 67%|██████▋   | 92/138 [4:45:53<2:21:08, 184.10s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3527
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 81663.86 | bwd_microstep: 89811.95 | bwd_inner_microstep: 88578.49 | bwd_allreduce_microstep: 654.08 | step_microstep: 20.35
[Rank 0] time (ms) | fwd: 81685.49 | bwd: 89889.71 | bwd_inner: 88760.87 | bwd_allreduce: 633.60 | step: 20.35

 67%|██████▋   | 93/138 [4:49:05<2:19:53, 186.52s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3539
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 84321.99 | bwd_microstep: 89556.05 | bwd_inner_microstep: 88672.23 | bwd_allreduce_microstep: 368.62 | step_microstep: 19.23
[Rank 0] time (ms) | fwd: 84329.00 | bwd: 89628.00 | bwd_inner: 88831.14 | bwd_allreduce: 386.71 | step: 19.23

 68%|██████▊   | 94/138 [4:52:18<2:18:15, 188.54s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 79530.65 | bwd_microstep: 87010.67 | bwd_inner_microstep: 85998.13 | bwd_allreduce_microstep: 552.99 | step_microstep: 27.05
[Rank 0] time (ms) | fwd: 79544.56 | bwd: 87082.41 | bwd_inner: 86148.61 | bwd_allreduce: 539.80 | step: 27.03

 69%|██████▉   | 95/138 [4:55:26<2:14:52, 188.20s/it]
                                                     
{'loss': 0.0117, 'grad_norm': 0.002933693816885352, 'learning_rate': 4.729772884265212e-05, 'epoch': 0.69}

 69%|██████▉   | 95/138 [4:55:26<2:14:52, 188.20s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 80112.59 | bwd_microstep: 85681.54 | bwd_inner_microstep: 84671.79 | bwd_allreduce_microstep: 603.71 | step_microstep: 19.37
[Rank 0] time (ms) | fwd: 80128.86 | bwd: 85742.95 | bwd_inner: 84823.24 | bwd_allreduce: 607.64 | step: 19.39

 70%|██████▉   | 96/138 [4:58:32<2:11:24, 187.72s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 77821.08 | bwd_microstep: 83237.30 | bwd_inner_microstep: 82492.71 | bwd_allreduce_microstep: 336.97 | step_microstep: 21.56
[Rank 0] time (ms) | fwd: 77813.75 | bwd: 83295.84 | bwd_inner: 82648.92 | bwd_allreduce: 342.33 | step: 21.60

 70%|███████   | 97/138 [5:01:35<2:07:18, 186.30s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.49 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 78252.80 | bwd_microstep: 82682.26 | bwd_inner_microstep: 81597.23 | bwd_allreduce_microstep: 689.45 | step_microstep: 23.14
[Rank 0] time (ms) | fwd: 78249.47 | bwd: 82759.47 | bwd_inner: 81710.00 | bwd_allreduce: 681.70 | step: 23.14

 71%|███████   | 98/138 [5:04:38<2:03:32, 185.30s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 79685.53 | bwd_microstep: 85614.63 | bwd_inner_microstep: 84596.07 | bwd_allreduce_microstep: 512.42 | step_microstep: 21.17
[Rank 0] time (ms) | fwd: 79692.86 | bwd: 85682.89 | bwd_inner: 84769.36 | bwd_allreduce: 475.66 | step: 21.17

 72%|███████▏  | 99/138 [5:07:45<2:00:39, 185.62s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 78504.65 | bwd_microstep: 84493.89 | bwd_inner_microstep: 83557.88 | bwd_allreduce_microstep: 479.79 | step_microstep: 19.57
[Rank 0] time (ms) | fwd: 78523.45 | bwd: 84571.95 | bwd_inner: 83702.33 | bwd_allreduce: 478.77 | step: 19.57

 72%|███████▏  | 100/138 [5:10:48<1:57:11, 185.04s/it]
                                                      
{'loss': 0.0083, 'grad_norm': 0.002227836288511753, 'learning_rate': 3.7651019814126654e-05, 'epoch': 0.72}

 72%|███████▏  | 100/138 [5:10:48<1:57:11, 185.04s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 82779.96 | bwd_microstep: 91012.30 | bwd_inner_microstep: 90181.43 | bwd_allreduce_microstep: 356.71 | step_microstep: 18.31
[Rank 0] time (ms) | fwd: 82818.89 | bwd: 91077.27 | bwd_inner: 90361.11 | bwd_allreduce: 356.34 | step: 18.32

 73%|███████▎  | 101/138 [5:14:01<1:55:32, 187.36s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3540
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 82641.77 | bwd_microstep: 87856.69 | bwd_inner_microstep: 86765.23 | bwd_allreduce_microstep: 655.88 | step_microstep: 19.77
[Rank 0] time (ms) | fwd: 82649.28 | bwd: 87924.62 | bwd_inner: 86912.73 | bwd_allreduce: 587.45 | step: 19.77

 74%|███████▍  | 102/138 [5:17:12<1:53:07, 188.55s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 78962.04 | bwd_microstep: 86861.23 | bwd_inner_microstep: 85598.27 | bwd_allreduce_microstep: 577.39 | step_microstep: 19.40
[Rank 0] time (ms) | fwd: 78965.51 | bwd: 86943.38 | bwd_inner: 85961.81 | bwd_allreduce: 581.33 | step: 19.41

 75%|███████▍  | 103/138 [5:20:21<1:50:00, 188.59s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 80269.91 | bwd_microstep: 89219.80 | bwd_inner_microstep: 87978.26 | bwd_allreduce_microstep: 640.36 | step_microstep: 20.60
[Rank 0] time (ms) | fwd: 80280.52 | bwd: 89308.73 | bwd_inner: 88209.14 | bwd_allreduce: 626.47 | step: 20.63

 75%|███████▌  | 104/138 [5:23:31<1:47:07, 189.05s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 79514.01 | bwd_microstep: 85672.33 | bwd_inner_microstep: 84718.73 | bwd_allreduce_microstep: 485.54 | step_microstep: 21.61
[Rank 0] time (ms) | fwd: 79529.70 | bwd: 85756.75 | bwd_inner: 84870.99 | bwd_allreduce: 475.04 | step: 21.59

 76%|███████▌  | 105/138 [5:26:36<1:43:14, 187.71s/it]
                                                      
{'loss': 0.0101, 'grad_norm': 0.0033331240992993116, 'learning_rate': 2.8872993029040508e-05, 'epoch': 0.76}

 76%|███████▌  | 105/138 [5:26:36<1:43:14, 187.71s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 77470.56 | bwd_microstep: 84886.68 | bwd_inner_microstep: 84018.34 | bwd_allreduce_microstep: 458.32 | step_microstep: 20.23
[Rank 0] time (ms) | fwd: 77478.60 | bwd: 84961.00 | bwd_inner: 84164.34 | bwd_allreduce: 469.37 | step: 20.23

 77%|███████▋  | 106/138 [5:29:40<1:39:32, 186.63s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.35 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 79884.33 | bwd_microstep: 86558.06 | bwd_inner_microstep: 85645.90 | bwd_allreduce_microstep: 477.18 | step_microstep: 19.20
[Rank 0] time (ms) | fwd: 79879.72 | bwd: 86630.81 | bwd_inner: 85787.30 | bwd_allreduce: 481.17 | step: 19.21

 78%|███████▊  | 107/138 [5:32:49<1:36:45, 187.28s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 79444.53 | bwd_microstep: 87316.15 | bwd_inner_microstep: 86255.35 | bwd_allreduce_microstep: 546.44 | step_microstep: 20.71
[Rank 0] time (ms) | fwd: 79456.17 | bwd: 87396.27 | bwd_inner: 86431.03 | bwd_allreduce: 511.79 | step: 20.71

 78%|███████▊  | 108/138 [5:35:57<1:33:43, 187.47s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.35 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 79566.47 | bwd_microstep: 86997.80 | bwd_inner_microstep: 86199.91 | bwd_allreduce_microstep: 376.93 | step_microstep: 21.55
[Rank 0] time (ms) | fwd: 79581.37 | bwd: 87072.03 | bwd_inner: 86339.17 | bwd_allreduce: 380.95 | step: 21.55

 79%|███████▉  | 109/138 [5:39:04<1:30:36, 187.45s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 79144.56 | bwd_microstep: 84832.79 | bwd_inner_microstep: 83698.97 | bwd_allreduce_microstep: 582.94 | step_microstep: 18.72
[Rank 0] time (ms) | fwd: 79155.19 | bwd: 84890.53 | bwd_inner: 83915.88 | bwd_allreduce: 594.86 | step: 18.72

 80%|███████▉  | 110/138 [5:42:09<1:27:07, 186.68s/it]
                                                      
{'loss': 0.01, 'grad_norm': 0.00224496191367507, 'learning_rate': 2.1085949060360654e-05, 'epoch': 0.8}

 80%|███████▉  | 110/138 [5:42:09<1:27:07, 186.68s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.40 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 81963.49 | bwd_microstep: 86491.01 | bwd_inner_microstep: 85576.44 | bwd_allreduce_microstep: 486.32 | step_microstep: 25.70
[Rank 0] time (ms) | fwd: 81972.33 | bwd: 86550.80 | bwd_inner: 85722.68 | bwd_allreduce: 497.24 | step: 25.70

 80%|████████  | 111/138 [5:45:16<1:24:02, 186.77s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 79346.93 | bwd_microstep: 84168.15 | bwd_inner_microstep: 83201.62 | bwd_allreduce_microstep: 484.08 | step_microstep: 19.46
[Rank 0] time (ms) | fwd: 79361.45 | bwd: 84250.99 | bwd_inner: 83360.55 | bwd_allreduce: 486.98 | step: 19.46

 81%|████████  | 112/138 [5:48:20<1:20:33, 185.89s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.67 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 79532.01 | bwd_microstep: 87150.80 | bwd_inner_microstep: 86057.49 | bwd_allreduce_microstep: 589.86 | step_microstep: 32.38
[Rank 0] time (ms) | fwd: 79535.34 | bwd: 87221.10 | bwd_inner: 86225.79 | bwd_allreduce: 599.56 | step: 32.37

 82%|████████▏ | 113/138 [5:51:30<1:17:58, 187.13s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3528
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.45 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 83116.25 | bwd_microstep: 87856.28 | bwd_inner_microstep: 86946.40 | bwd_allreduce_microstep: 491.54 | step_microstep: 21.97
[Rank 0] time (ms) | fwd: 83146.50 | bwd: 87917.40 | bwd_inner: 87100.44 | bwd_allreduce: 489.97 | step: 21.97

 83%|████████▎ | 114/138 [5:54:41<1:15:22, 188.44s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3538
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.47 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 80373.07 | bwd_microstep: 88805.62 | bwd_inner_microstep: 87611.32 | bwd_allreduce_microstep: 611.73 | step_microstep: 27.65
[Rank 0] time (ms) | fwd: 80383.70 | bwd: 88880.31 | bwd_inner: 87848.54 | bwd_allreduce: 625.08 | step: 27.63

 83%|████████▎ | 115/138 [5:57:53<1:12:40, 189.58s/it]
                                                      
{'loss': 0.0085, 'grad_norm': 0.002678320510312915, 'learning_rate': 1.439838153227e-05, 'epoch': 0.83}

 83%|████████▎ | 115/138 [5:57:53<1:12:40, 189.58s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3447
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 81308.62 | bwd_microstep: 86547.64 | bwd_inner_microstep: 85543.44 | bwd_allreduce_microstep: 575.47 | step_microstep: 23.10
[Rank 0] time (ms) | fwd: 81316.94 | bwd: 86605.73 | bwd_inner: 85716.86 | bwd_allreduce: 589.15 | step: 23.11

 84%|████████▍ | 116/138 [6:01:01<1:09:18, 189.02s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.49 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 78130.32 | bwd_microstep: 84850.35 | bwd_inner_microstep: 83686.32 | bwd_allreduce_microstep: 591.04 | step_microstep: 20.03
[Rank 0] time (ms) | fwd: 78156.35 | bwd: 84947.15 | bwd_inner: 83834.50 | bwd_allreduce: 612.78 | step: 20.03

 85%|████████▍ | 117/138 [6:04:05<1:05:37, 187.51s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 77552.10 | bwd_microstep: 85761.47 | bwd_inner_microstep: 84699.85 | bwd_allreduce_microstep: 499.67 | step_microstep: 19.85
[Rank 0] time (ms) | fwd: 77559.31 | bwd: 85820.64 | bwd_inner: 84887.12 | bwd_allreduce: 489.44 | step: 19.86

 86%|████████▌ | 118/138 [6:07:11<1:02:19, 187.00s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 80860.17 | bwd_microstep: 83909.94 | bwd_inner_microstep: 82902.76 | bwd_allreduce_microstep: 564.17 | step_microstep: 19.45
[Rank 0] time (ms) | fwd: 80886.99 | bwd: 83970.57 | bwd_inner: 83017.82 | bwd_allreduce: 603.96 | step: 19.46

 86%|████████▌ | 119/138 [6:10:17<59:05, 186.61s/it]  dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 81562.61 | bwd_microstep: 87203.12 | bwd_inner_microstep: 86276.81 | bwd_allreduce_microstep: 530.93 | step_microstep: 19.08
[Rank 0] time (ms) | fwd: 81592.53 | bwd: 87271.29 | bwd_inner: 86397.93 | bwd_allreduce: 542.65 | step: 19.09

 87%|████████▋ | 120/138 [6:13:26<56:11, 187.29s/it]
                                                    
{'loss': 0.0112, 'grad_norm': 0.001902088988572359, 'learning_rate': 8.903465523913957e-06, 'epoch': 0.87}

 87%|████████▋ | 120/138 [6:13:26<56:11, 187.29s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3536
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 79647.67 | bwd_microstep: 87795.71 | bwd_inner_microstep: 86855.08 | bwd_allreduce_microstep: 490.58 | step_microstep: 23.73
[Rank 0] time (ms) | fwd: 79658.67 | bwd: 87865.55 | bwd_inner: 86992.43 | bwd_allreduce: 498.25 | step: 23.73

 88%|████████▊ | 121/138 [6:16:34<53:10, 187.70s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 79043.40 | bwd_microstep: 86510.09 | bwd_inner_microstep: 85323.55 | bwd_allreduce_microstep: 605.92 | step_microstep: 19.96
[Rank 0] time (ms) | fwd: 79070.03 | bwd: 86597.18 | bwd_inner: 85522.91 | bwd_allreduce: 610.34 | step: 19.97

 88%|████████▊ | 122/138 [6:19:38<49:46, 186.66s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 16, images per sample: 8.0, dynamic token length: 3535
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3535
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 82303.00 | bwd_microstep: 88627.42 | bwd_inner_microstep: 87613.10 | bwd_allreduce_microstep: 510.72 | step_microstep: 20.52
[Rank 0] time (ms) | fwd: 82315.36 | bwd: 88703.70 | bwd_inner: 87760.89 | bwd_allreduce: 512.13 | step: 20.54

 89%|████████▉ | 123/138 [6:22:50<46:59, 187.99s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3535
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3533
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.24 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 83154.01 | bwd_microstep: 87398.68 | bwd_inner_microstep: 86500.03 | bwd_allreduce_microstep: 492.71 | step_microstep: 30.14
[Rank 0] time (ms) | fwd: 83166.35 | bwd: 87459.96 | bwd_inner: 86630.31 | bwd_allreduce: 502.43 | step: 30.16

 90%|████████▉ | 124/138 [6:26:00<44:03, 188.81s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 80104.70 | bwd_microstep: 86378.22 | bwd_inner_microstep: 85344.87 | bwd_allreduce_microstep: 505.46 | step_microstep: 20.09
[Rank 0] time (ms) | fwd: 80112.14 | bwd: 86460.81 | bwd_inner: 85501.75 | bwd_allreduce: 511.41 | step: 20.08

 91%|█████████ | 125/138 [6:29:06<40:43, 187.93s/it]
                                                    
{'loss': 0.0126, 'grad_norm': 0.003356203204020858, 'learning_rate': 4.6777594000230855e-06, 'epoch': 0.9}

 91%|█████████ | 125/138 [6:29:06<40:43, 187.93s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3468
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.73
[Rank 0] time (ms) | fwd_microstep: 80827.82 | bwd_microstep: 86706.66 | bwd_inner_microstep: 85635.99 | bwd_allreduce_microstep: 567.27 | step_microstep: 17.95
[Rank 0] time (ms) | fwd: 80832.85 | bwd: 86789.92 | bwd_inner: 85792.89 | bwd_allreduce: 594.42 | step: 17.97

 91%|█████████▏| 126/138 [6:32:14<37:35, 187.95s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3526
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.74 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 81169.63 | bwd_microstep: 88292.52 | bwd_inner_microstep: 86070.21 | bwd_allreduce_microstep: 1724.47 | step_microstep: 28.32
[Rank 0] time (ms) | fwd: 81177.27 | bwd: 88384.11 | bwd_inner: 86230.48 | bwd_allreduce: 1746.56 | step: 28.30

 92%|█████████▏| 127/138 [6:35:24<34:32, 188.43s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 77888.26 | bwd_microstep: 86114.93 | bwd_inner_microstep: 84958.82 | bwd_allreduce_microstep: 676.47 | step_microstep: 21.02
[Rank 0] time (ms) | fwd: 77893.22 | bwd: 86184.04 | bwd_inner: 85122.15 | bwd_allreduce: 685.86 | step: 21.03

 93%|█████████▎| 128/138 [6:38:30<31:18, 187.80s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 20, images per sample: 10.0, dynamic token length: 3537
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 80610.67 | bwd_microstep: 86212.20 | bwd_inner_microstep: 84835.42 | bwd_allreduce_microstep: 717.30 | step_microstep: 19.78
[Rank 0] time (ms) | fwd: 80612.47 | bwd: 86277.54 | bwd_inner: 84975.56 | bwd_allreduce: 930.91 | step: 19.79

 93%|█████████▎| 129/138 [6:41:37<28:08, 187.62s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 78095.88 | bwd_microstep: 84791.95 | bwd_inner_microstep: 83911.33 | bwd_allreduce_microstep: 407.32 | step_microstep: 21.76
[Rank 0] time (ms) | fwd: 78103.68 | bwd: 84861.50 | bwd_inner: 84062.87 | bwd_allreduce: 425.86 | step: 21.76

 94%|█████████▍| 130/138 [6:44:42<24:55, 186.89s/it]
                                                    
{'loss': 0.0109, 'grad_norm': 0.002311143558472395, 'learning_rate': 1.7801381552624563e-06, 'epoch': 0.94}

 94%|█████████▍| 130/138 [6:44:42<24:55, 186.89s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.58 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 81663.69 | bwd_microstep: 88832.53 | bwd_inner_microstep: 87492.02 | bwd_allreduce_microstep: 740.28 | step_microstep: 39.46
[Rank 0] time (ms) | fwd: 81685.15 | bwd: 88910.39 | bwd_inner: 87703.96 | bwd_allreduce: 752.60 | step: 39.36

 95%|█████████▍| 131/138 [6:47:50<21:50, 187.21s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 79640.98 | bwd_microstep: 88758.99 | bwd_inner_microstep: 87727.27 | bwd_allreduce_microstep: 569.14 | step_microstep: 19.19
[Rank 0] time (ms) | fwd: 79682.12 | bwd: 88833.41 | bwd_inner: 87879.38 | bwd_allreduce: 578.31 | step: 19.19

 96%|█████████▌| 132/138 [6:51:00<18:48, 188.04s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.53 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 79892.77 | bwd_microstep: 87644.28 | bwd_inner_microstep: 86630.10 | bwd_allreduce_microstep: 526.87 | step_microstep: 24.16
[Rank 0] time (ms) | fwd: 79928.89 | bwd: 87723.05 | bwd_inner: 86800.41 | bwd_allreduce: 547.36 | step: 24.16

 96%|█████████▋| 133/138 [6:54:08<15:39, 187.89s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 79139.56 | bwd_microstep: 85148.91 | bwd_inner_microstep: 84076.41 | bwd_allreduce_microstep: 598.54 | step_microstep: 20.33
[Rank 0] time (ms) | fwd: 79156.88 | bwd: 85207.50 | bwd_inner: 84241.34 | bwd_allreduce: 596.16 | step: 20.34

 97%|█████████▋| 134/138 [6:57:13<12:28, 187.01s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.64 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 77851.35 | bwd_microstep: 85214.48 | bwd_inner_microstep: 84057.14 | bwd_allreduce_microstep: 709.52 | step_microstep: 22.15
[Rank 0] time (ms) | fwd: 77854.48 | bwd: 85309.48 | bwd_inner: 84207.84 | bwd_allreduce: 691.42 | step: 22.15

 98%|█████████▊| 135/138 [7:00:17<09:18, 186.28s/it]
                                                    
{'loss': 0.0102, 'grad_norm': 0.002950450871139765, 'learning_rate': 2.509731335744281e-07, 'epoch': 0.98}

 98%|█████████▊| 135/138 [7:00:17<09:18, 186.28s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.45 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 79229.90 | bwd_microstep: 83762.43 | bwd_inner_microstep: 82803.11 | bwd_allreduce_microstep: 419.32 | step_microstep: 20.72
[Rank 0] time (ms) | fwd: 79235.24 | bwd: 83817.08 | bwd_inner: 82938.04 | bwd_allreduce: 399.22 | step: 20.72

 99%|█████████▊| 136/138 [7:03:22<06:11, 185.69s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 10, images per sample: 5.0, dynamic token length: 1932
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 78212.92 | bwd_microstep: 84522.67 | bwd_inner_microstep: 83355.89 | bwd_allreduce_microstep: 665.61 | step_microstep: 20.89
[Rank 0] time (ms) | fwd: 78218.18 | bwd: 84611.55 | bwd_inner: 83504.62 | bwd_allreduce: 671.59 | step: 20.90

 99%|█████████▉| 137/138 [7:06:26<03:05, 185.18s/it]dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 14, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.17 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 79575.92 | bwd_microstep: 87589.15 | bwd_inner_microstep: 84836.72 | bwd_allreduce_microstep: 2195.47 | step_microstep: 38.73
[Rank 0] time (ms) | fwd: 79591.80 | bwd: 87676.22 | bwd_inner: 85004.20 | bwd_allreduce: 2214.37 | step: 38.67

100%|██████████| 138/138 [7:09:32<00:00, 185.65s/it]petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[INFO|trainer.py:2329] 2026-07-03 21:38:54,284 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



                                                    
{'train_runtime': 25778.5856, 'train_samples_per_second': 0.343, 'train_steps_per_second': 0.005, 'train_loss': 0.02036970402991426, 'epoch': 1.0}

100%|██████████| 138/138 [7:09:35<00:00, 185.65s/it]
100%|██████████| 138/138 [7:09:35<00:00, 186.78s/it]
[INFO|trainer.py:3410] 2026-07-03 21:38:59,505 >> Saving model checkpoint to /root/ai-project/Vintern/vlm_training
[INFO|configuration_utils.py:472] 2026-07-03 21:38:59,515 >> Configuration saved in /root/ai-project/Vintern/vlm_training/config.json
[INFO|configuration_utils.py:731] 2026-07-03 21:38:59,517 >> Configuration saved in /root/ai-project/Vintern/vlm_training/generation_config.json
[INFO|modeling_utils.py:2618] 2026-07-03 21:39:02,139 >> Model weights saved in /root/ai-project/Vintern/vlm_training/model.safetensors
[INFO|tokenization_utils_base.py:2513] 2026-07-03 21:39:02,161 >> tokenizer config file saved in /root/ai-project/Vintern/vlm_training/tokenizer_config.json
[INFO|tokenization_utils_base.py:2522] 2026-07-03 21:39:02,163 >> Special tokens file saved in /root/ai-project/Vintern/vlm_training/special_tokens_map.json
[INFO|tokenization_utils_base.py:2573] 2026-07-03 21:39:02,164 >> added tokens file saved in /root/ai-project/Vintern/vlm_training/added_tokens.json
***** train metrics *****
  epoch                    =         0.9986
  total_flos               = 169354287872GF
  train_loss               =         0.0204
  train_runtime            =     7:09:38.58
  train_samples            =           8843
  train_samples_per_second =          0.343
  train_steps_per_second   =          0.005
wandb: 
wandb: 🚀 View run vintern_lora_card at: https://wandb.ai/hhung26062000/huggingface/runs/c0vco66y
wandb: Find logs at: wandb/run-20260703_142916-c0vco66y/logs
[rank0]:[W703 21:39:06.679979361 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/06/2026 09:33:48 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/06/2026 09:33:48 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=64,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul06_09-33-48_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=5,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=1.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/06/2026 09:33:48 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-06 09:33:48,285 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 09:33:48,285 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-06 09:33:48,285 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 09:33:48,285 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 09:33:48,285 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 09:33:48,285 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-06 09:33:48,647 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/06/2026 09:33:48 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-06 09:33:48,658 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-06 09:33:48,660 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/06/2026 09:33:48 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-06 09:33:48,661 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-06 09:33:48,692 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-06 09:33:48,694 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-06 09:33:48,753 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-06 09:33:49,835 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-06 09:33:49,836 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-06 09:33:49,841 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-06 09:33:49,842 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/06/2026 09:33:49 - INFO - __main__ - Finished
07/06/2026 09:33:49 - INFO - __main__ - model.config.force_image_size: 448
07/06/2026 09:33:49 - INFO - __main__ - data_args.force_image_size: 448
07/06/2026 09:33:49 - INFO - __main__ - model.config.vision_config.image_size: 448
07/06/2026 09:33:49 - INFO - __main__ - max_dynamic_patch is set to 12 according to the meta file
07/06/2026 09:33:49 - INFO - __main__ - [Dataset] num_image_token: 256
07/06/2026 09:33:49 - INFO - __main__ - [Dataset] dynamic_image_size: True
07/06/2026 09:33:49 - INFO - __main__ - [Dataset] use_thumbnail: True
07/06/2026 09:33:49 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 12
07/06/2026 09:33:49 - INFO - __main__ - Formatting inputs...Skip in lazy mode
07/06/2026 09:33:50 - INFO - __main__ - Add dataset: vi-envelope-vqa with length: 8843
trainable params: 8,798,208 || all params: 638,496,128 || trainable%: 1.3779579255334184
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight
07/06/2026 09:33:51 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight
07/06/2026 09:33:51 - INFO - __main__ - mlp1.0.weight
07/06/2026 09:33:51 - INFO - __main__ - mlp1.0.bias
07/06/2026 09:33:51 - INFO - __main__ - mlp1.1.weight
07/06/2026 09:33:51 - INFO - __main__ - mlp1.1.bias
07/06/2026 09:33:51 - INFO - __main__ - mlp1.3.weight
07/06/2026 09:33:51 - INFO - __main__ - mlp1.3.bias
[INFO|trainer.py:641] 2026-07-06 09:33:51,077 >> Using auto half precision backend
[INFO|trainer.py:2078] 2026-07-06 09:33:52,321 >> ***** Running training *****
[INFO|trainer.py:2079] 2026-07-06 09:33:52,321 >>   Num examples = 8,843
[INFO|trainer.py:2080] 2026-07-06 09:33:52,321 >>   Num Epochs = 1
[INFO|trainer.py:2081] 2026-07-06 09:33:52,321 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2084] 2026-07-06 09:33:52,321 >>   Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:2085] 2026-07-06 09:33:52,321 >>   Gradient Accumulation steps = 64
[INFO|trainer.py:2086] 2026-07-06 09:33:52,321 >>   Total optimization steps = 138
[INFO|trainer.py:2087] 2026-07-06 09:33:52,327 >>   Number of trainable parameters = 13,281,024
[INFO|integration_utils.py:723] 2026-07-06 09:33:52,332 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
wandb: Currently logged in as: hhung2662000 (hhung26062000) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in /root/ai-project/Vintern/internvl_chat/wandb/run-20260706_093353-ogttutpj
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run vintern_lora_card
wandb: ⭐️ View project at https://wandb.ai/hhung26062000/huggingface
wandb: 🚀 View run at https://wandb.ai/hhung26062000/huggingface/runs/ogttutpj

  0%|          | 0/138 [00:00<?, ?it/s]/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.25 | optimizer_gradients: 28.88 | optimizer_step: 16.45
[Rank 0] time (ms) | fwd_microstep: 92576.24 | bwd_microstep: 98620.61 | bwd_inner_microstep: 98346.45 | bwd_allreduce_microstep: 78.72 | step_microstep: 142.34
[Rank 0] time (ms) | fwd: 92579.87 | bwd: 98590.51 | bwd_inner: 98333.95 | bwd_allreduce: 75.59 | step: 142.39

  1%|          | 1/138 [03:53<8:54:16, 233.99s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 94023.46 | bwd_microstep: 98816.85 | bwd_inner_microstep: 98505.45 | bwd_allreduce_microstep: 106.71 | step_microstep: 19.14
[Rank 0] time (ms) | fwd: 94021.07 | bwd: 98799.69 | bwd_inner: 98524.54 | bwd_allreduce: 104.37 | step: 19.15

  1%|▏         | 2/138 [07:17<8:09:07, 215.79s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 93942.75 | bwd_microstep: 98878.90 | bwd_inner_microstep: 98580.86 | bwd_allreduce_microstep: 100.34 | step_microstep: 20.25
[Rank 0] time (ms) | fwd: 93941.01 | bwd: 98860.52 | bwd_inner: 98586.70 | bwd_allreduce: 103.04 | step: 20.28

  2%|▏         | 3/138 [10:39<7:52:18, 209.92s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93767.27 | bwd_microstep: 100842.86 | bwd_inner_microstep: 100631.99 | bwd_allreduce_microstep: 71.92 | step_microstep: 17.32
[Rank 0] time (ms) | fwd: 93765.54 | bwd: 100833.76 | bwd_inner: 100631.50 | bwd_allreduce: 74.98 | step: 17.33

  3%|▎         | 4/138 [14:01<7:41:46, 206.76s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 93688.45 | bwd_microstep: 102377.35 | bwd_inner_microstep: 102135.20 | bwd_allreduce_microstep: 76.46 | step_microstep: 18.45
[Rank 0] time (ms) | fwd: 93687.05 | bwd: 102356.67 | bwd_inner: 102127.30 | bwd_allreduce: 75.05 | step: 18.46

  4%|▎         | 5/138 [17:23<7:34:03, 204.84s/it]
                                                  
{'loss': 0.2382, 'grad_norm': 0.012892262078821659, 'learning_rate': 0.0002, 'epoch': 0.04}

  4%|▎         | 5/138 [17:23<7:34:03, 204.84s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 94008.68 | bwd_microstep: 101390.57 | bwd_inner_microstep: 101084.23 | bwd_allreduce_microstep: 116.10 | step_microstep: 20.62
[Rank 0] time (ms) | fwd: 94007.21 | bwd: 101386.07 | bwd_inner: 101089.91 | bwd_allreduce: 110.57 | step: 20.64

  4%|▍         | 6/138 [20:45<7:28:39, 203.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93532.04 | bwd_microstep: 99848.35 | bwd_inner_microstep: 99403.67 | bwd_allreduce_microstep: 123.37 | step_microstep: 18.08
[Rank 0] time (ms) | fwd: 93530.48 | bwd: 99835.40 | bwd_inner: 99460.81 | bwd_allreduce: 143.88 | step: 18.09

  5%|▌         | 7/138 [24:06<7:23:26, 203.11s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 92695.33 | bwd_microstep: 99309.78 | bwd_inner_microstep: 99067.64 | bwd_allreduce_microstep: 79.11 | step_microstep: 17.93
[Rank 0] time (ms) | fwd: 92693.69 | bwd: 99306.18 | bwd_inner: 99083.99 | bwd_allreduce: 84.84 | step: 17.93

  6%|▌         | 8/138 [27:27<7:18:08, 202.22s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 93587.26 | bwd_microstep: 99304.76 | bwd_inner_microstep: 98894.67 | bwd_allreduce_microstep: 152.93 | step_microstep: 17.97
[Rank 0] time (ms) | fwd: 93585.54 | bwd: 99298.17 | bwd_inner: 98946.98 | bwd_allreduce: 155.60 | step: 17.98

  7%|▋         | 9/138 [30:48<7:14:09, 201.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 92739.49 | bwd_microstep: 98095.33 | bwd_inner_microstep: 97774.11 | bwd_allreduce_microstep: 95.55 | step_microstep: 17.65
[Rank 0] time (ms) | fwd: 92737.88 | bwd: 98063.77 | bwd_inner: 97772.97 | bwd_allreduce: 87.68 | step: 17.68

  7%|▋         | 10/138 [34:09<7:10:06, 201.62s/it]
                                                   
{'loss': 0.0396, 'grad_norm': 0.0053918142803013325, 'learning_rate': 0.00019930337092856243, 'epoch': 0.07}

  7%|▋         | 10/138 [34:09<7:10:06, 201.62s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94107.39 | bwd_microstep: 100590.74 | bwd_inner_microstep: 100405.79 | bwd_allreduce_microstep: 31.86 | step_microstep: 18.06
[Rank 0] time (ms) | fwd: 94105.74 | bwd: 100554.29 | bwd_inner: 100382.97 | bwd_allreduce: 34.40 | step: 18.07

  8%|▊         | 11/138 [37:32<7:07:33, 202.00s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 92284.15 | bwd_microstep: 98833.39 | bwd_inner_microstep: 98611.89 | bwd_allreduce_microstep: 63.55 | step_microstep: 19.80
[Rank 0] time (ms) | fwd: 92282.33 | bwd: 98818.89 | bwd_inner: 98606.61 | bwd_allreduce: 63.18 | step: 19.81

  9%|▊         | 12/138 [40:51<7:02:20, 201.11s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 92726.93 | bwd_microstep: 97457.63 | bwd_inner_microstep: 97287.80 | bwd_allreduce_microstep: 66.76 | step_microstep: 18.19
[Rank 0] time (ms) | fwd: 92725.38 | bwd: 97454.37 | bwd_inner: 97299.30 | bwd_allreduce: 60.84 | step: 18.20

  9%|▉         | 13/138 [44:11<6:58:08, 200.71s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2022
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.80 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94497.83 | bwd_microstep: 99437.59 | bwd_inner_microstep: 98899.24 | bwd_allreduce_microstep: 331.23 | step_microstep: 33.45
[Rank 0] time (ms) | fwd: 94496.20 | bwd: 99426.51 | bwd_inner: 98924.65 | bwd_allreduce: 330.68 | step: 33.50

 10%|█         | 14/138 [47:35<6:56:56, 201.75s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94887.82 | bwd_microstep: 101478.63 | bwd_inner_microstep: 101269.21 | bwd_allreduce_microstep: 47.29 | step_microstep: 17.99
[Rank 0] time (ms) | fwd: 94882.74 | bwd: 101436.53 | bwd_inner: 101226.46 | bwd_allreduce: 42.63 | step: 18.00

 11%|█         | 15/138 [50:59<6:55:12, 202.54s/it]
                                                   
{'loss': 0.0252, 'grad_norm': 0.0037580234929919243, 'learning_rate': 0.00019722318955551306, 'epoch': 0.11}

 11%|█         | 15/138 [50:59<6:55:12, 202.54s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93384.66 | bwd_microstep: 97916.02 | bwd_inner_microstep: 97709.11 | bwd_allreduce_microstep: 12.34 | step_microstep: 18.54
[Rank 0] time (ms) | fwd: 93382.73 | bwd: 97852.16 | bwd_inner: 97660.95 | bwd_allreduce: 12.57 | step: 18.54

 12%|█▏        | 16/138 [54:22<6:51:59, 202.62s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.29 | optimizer_step: 1.04
[Rank 0] time (ms) | fwd_microstep: 94445.07 | bwd_microstep: 100987.09 | bwd_inner_microstep: 100831.08 | bwd_allreduce_microstep: 19.89 | step_microstep: 30.44
[Rank 0] time (ms) | fwd: 94437.04 | bwd: 100941.44 | bwd_inner: 100773.36 | bwd_allreduce: 19.93 | step: 30.45

 12%|█▏        | 17/138 [57:48<6:50:53, 203.75s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 94970.82 | bwd_microstep: 101289.44 | bwd_inner_microstep: 101016.83 | bwd_allreduce_microstep: 54.65 | step_microstep: 17.23
[Rank 0] time (ms) | fwd: 94968.88 | bwd: 101237.35 | bwd_inner: 100993.12 | bwd_allreduce: 56.94 | step: 17.23

 13%|█▎        | 18/138 [1:01:14<6:48:28, 204.23s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.14 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 92607.51 | bwd_microstep: 98777.88 | bwd_inner_microstep: 98384.94 | bwd_allreduce_microstep: 136.74 | step_microstep: 17.18
[Rank 0] time (ms) | fwd: 92605.78 | bwd: 98787.36 | bwd_inner: 98443.76 | bwd_allreduce: 140.72 | step: 17.19

 14%|█▍        | 19/138 [1:04:35<6:43:07, 203.26s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.10 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 94373.66 | bwd_microstep: 103954.89 | bwd_inner_microstep: 103616.33 | bwd_allreduce_microstep: 90.86 | step_microstep: 17.32
[Rank 0] time (ms) | fwd: 94372.24 | bwd: 103932.05 | bwd_inner: 103631.74 | bwd_allreduce: 91.08 | step: 17.31

 14%|█▍        | 20/138 [1:07:59<6:40:21, 203.58s/it]
                                                     
{'loss': 0.0158, 'grad_norm': 0.0017777378670871258, 'learning_rate': 0.00019378843817721854, 'epoch': 0.14}

 14%|█▍        | 20/138 [1:07:59<6:40:21, 203.58s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3550
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 93737.21 | bwd_microstep: 102116.55 | bwd_inner_microstep: 101718.25 | bwd_allreduce_microstep: 185.36 | step_microstep: 19.74
[Rank 0] time (ms) | fwd: 93735.61 | bwd: 102100.58 | bwd_inner: 101755.71 | bwd_allreduce: 186.43 | step: 19.74

 15%|█▌        | 21/138 [1:11:23<6:37:04, 203.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.39 | optimizer_step: 1.00
[Rank 0] time (ms) | fwd_microstep: 92700.36 | bwd_microstep: 100609.55 | bwd_inner_microstep: 100234.55 | bwd_allreduce_microstep: 138.31 | step_microstep: 38.25
[Rank 0] time (ms) | fwd: 92698.99 | bwd: 100583.95 | bwd_inner: 100263.19 | bwd_allreduce: 136.64 | step: 38.25

 16%|█▌        | 22/138 [1:14:43<6:31:35, 202.55s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 95147.89 | bwd_microstep: 101077.76 | bwd_inner_microstep: 100760.68 | bwd_allreduce_microstep: 119.18 | step_microstep: 17.45
[Rank 0] time (ms) | fwd: 95146.06 | bwd: 101078.66 | bwd_inner: 100789.94 | bwd_allreduce: 117.97 | step: 17.45

 17%|█▋        | 23/138 [1:18:08<6:30:00, 203.49s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.15 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 91835.92 | bwd_microstep: 98203.00 | bwd_inner_microstep: 97889.17 | bwd_allreduce_microstep: 122.53 | step_microstep: 37.52
[Rank 0] time (ms) | fwd: 91834.35 | bwd: 98203.14 | bwd_inner: 97910.89 | bwd_allreduce: 128.50 | step: 37.56

 17%|█▋        | 24/138 [1:21:27<6:23:50, 202.03s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 92551.20 | bwd_microstep: 100134.18 | bwd_inner_microstep: 99875.41 | bwd_allreduce_microstep: 64.49 | step_microstep: 18.03
[Rank 0] time (ms) | fwd: 92549.55 | bwd: 100103.86 | bwd_inner: 99865.73 | bwd_allreduce: 66.06 | step: 18.03

 18%|█▊        | 25/138 [1:24:48<6:20:01, 201.79s/it]
                                                     
{'loss': 0.0151, 'grad_norm': 0.0024384628050029278, 'learning_rate': 0.00018904697174694447, 'epoch': 0.18}

 18%|█▊        | 25/138 [1:24:48<6:20:01, 201.79s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 92886.00 | bwd_microstep: 99855.79 | bwd_inner_microstep: 99522.87 | bwd_allreduce_microstep: 119.52 | step_microstep: 16.95
[Rank 0] time (ms) | fwd: 92884.29 | bwd: 99856.72 | bwd_inner: 99535.29 | bwd_allreduce: 114.05 | step: 17.00

 19%|█▉        | 26/138 [1:28:09<6:16:02, 201.45s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.45 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 92816.17 | bwd_microstep: 98885.81 | bwd_inner_microstep: 98668.50 | bwd_allreduce_microstep: 72.10 | step_microstep: 18.10
[Rank 0] time (ms) | fwd: 92814.42 | bwd: 98877.83 | bwd_inner: 98667.65 | bwd_allreduce: 71.48 | step: 18.11

 20%|█▉        | 27/138 [1:31:30<6:12:20, 201.26s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.05 | optimizer_gradients: 3.40 | optimizer_step: 1.21
[Rank 0] time (ms) | fwd_microstep: 92719.02 | bwd_microstep: 102111.25 | bwd_inner_microstep: 101855.90 | bwd_allreduce_microstep: 103.93 | step_microstep: 27.85
[Rank 0] time (ms) | fwd: 92717.65 | bwd: 102117.14 | bwd_inner: 101891.51 | bwd_allreduce: 100.61 | step: 27.95

 20%|██        | 28/138 [1:34:50<6:08:33, 201.03s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.56 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94718.04 | bwd_microstep: 99929.15 | bwd_inner_microstep: 99536.39 | bwd_allreduce_microstep: 155.86 | step_microstep: 34.26
[Rank 0] time (ms) | fwd: 94716.38 | bwd: 99939.42 | bwd_inner: 99584.74 | bwd_allreduce: 160.81 | step: 34.29

 21%|██        | 29/138 [1:38:14<6:06:52, 201.95s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 92403.76 | bwd_microstep: 99854.39 | bwd_inner_microstep: 99515.10 | bwd_allreduce_microstep: 111.39 | step_microstep: 17.14
[Rank 0] time (ms) | fwd: 92402.17 | bwd: 99840.92 | bwd_inner: 99539.40 | bwd_allreduce: 112.83 | step: 17.15

 22%|██▏       | 30/138 [1:41:34<6:02:16, 201.27s/it]
                                                     
{'loss': 0.0112, 'grad_norm': 0.0018282835371792316, 'learning_rate': 0.0001830648511318223, 'epoch': 0.22}

 22%|██▏       | 30/138 [1:41:34<6:02:16, 201.27s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 93332.35 | bwd_microstep: 98730.62 | bwd_inner_microstep: 98302.27 | bwd_allreduce_microstep: 144.52 | step_microstep: 18.96
[Rank 0] time (ms) | fwd: 93330.53 | bwd: 98713.97 | bwd_inner: 98301.17 | bwd_allreduce: 147.18 | step: 18.97

 22%|██▏       | 31/138 [1:44:56<5:59:01, 201.33s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.24 | optimizer_gradients: 7.86 | optimizer_step: 1.87
[Rank 0] time (ms) | fwd_microstep: 92933.51 | bwd_microstep: 100445.83 | bwd_inner_microstep: 100047.71 | bwd_allreduce_microstep: 167.48 | step_microstep: 347.49
[Rank 0] time (ms) | fwd: 92931.60 | bwd: 100451.03 | bwd_inner: 100079.22 | bwd_allreduce: 168.24 | step: 347.51

 23%|██▎       | 32/138 [1:48:18<5:56:31, 201.81s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.05 | optimizer_gradients: 5.94 | optimizer_step: 1.46
[Rank 0] time (ms) | fwd_microstep: 95176.17 | bwd_microstep: 101530.21 | bwd_inner_microstep: 101191.12 | bwd_allreduce_microstep: 56.91 | step_microstep: 376.39
[Rank 0] time (ms) | fwd: 95173.59 | bwd: 101450.33 | bwd_inner: 101135.02 | bwd_allreduce: 61.45 | step: 376.39

 24%|██▍       | 33/138 [1:51:49<5:57:47, 204.46s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.19 | optimizer_gradients: 7.53 | optimizer_step: 1.71
[Rank 0] time (ms) | fwd_microstep: 98055.78 | bwd_microstep: 98672.51 | bwd_inner_microstep: 98485.35 | bwd_allreduce_microstep: 42.49 | step_microstep: 336.89
[Rank 0] time (ms) | fwd: 98049.06 | bwd: 98614.84 | bwd_inner: 98449.36 | bwd_allreduce: 42.66 | step: 336.89

 25%|██▍       | 34/138 [1:55:24<5:59:35, 207.46s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 96540.55 | bwd_microstep: 101151.50 | bwd_inner_microstep: 100966.37 | bwd_allreduce_microstep: 11.89 | step_microstep: 17.80
[Rank 0] time (ms) | fwd: 96535.70 | bwd: 101086.41 | bwd_inner: 100917.64 | bwd_allreduce: 11.93 | step: 17.81

 25%|██▌       | 35/138 [1:58:53<5:57:03, 208.00s/it]
                                                     
{'loss': 0.0107, 'grad_norm': 0.002051867777481675, 'learning_rate': 0.00017592542271443887, 'epoch': 0.25}

 25%|██▌       | 35/138 [1:58:53<5:57:03, 208.00s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.12 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 97920.55 | bwd_microstep: 103243.43 | bwd_inner_microstep: 102957.28 | bwd_allreduce_microstep: 82.79 | step_microstep: 38.10
[Rank 0] time (ms) | fwd: 97915.00 | bwd: 103196.87 | bwd_inner: 102926.36 | bwd_allreduce: 81.48 | step: 38.13

 26%|██▌       | 36/138 [2:02:30<5:58:05, 210.64s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.11 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 98509.63 | bwd_microstep: 104652.19 | bwd_inner_microstep: 104412.98 | bwd_allreduce_microstep: 77.29 | step_microstep: 16.72
[Rank 0] time (ms) | fwd: 98507.00 | bwd: 104632.97 | bwd_inner: 104417.13 | bwd_allreduce: 79.74 | step: 16.73

 27%|██▋       | 37/138 [2:06:05<5:57:00, 212.08s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 97592.39 | bwd_microstep: 98823.24 | bwd_inner_microstep: 98289.31 | bwd_allreduce_microstep: 345.36 | step_microstep: 24.88
[Rank 0] time (ms) | fwd: 97583.63 | bwd: 98779.11 | bwd_inner: 98244.28 | bwd_allreduce: 344.87 | step: 24.87

 28%|██▊       | 38/138 [2:09:38<5:53:53, 212.33s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
[Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 3.65 | optimizer_step: 0.96
[Rank 0] time (ms) | fwd_microstep: 95831.90 | bwd_microstep: 101318.84 | bwd_inner_microstep: 101131.55 | bwd_allreduce_microstep: 37.47 | step_microstep: 58.22
[Rank 0] time (ms) | fwd: 95829.40 | bwd: 101276.59 | bwd_inner: 101099.48 | bwd_allreduce: 41.25 | step: 58.25

 28%|██▊       | 39/138 [2:13:07<5:48:56, 211.48s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 969
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3534
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.53 | optimizer_step: 0.97
[Rank 0] time (ms) | fwd_microstep: 94703.27 | bwd_microstep: 101903.01 | bwd_inner_microstep: 101594.91 | bwd_allreduce_microstep: 96.03 | step_microstep: 46.21
[Rank 0] time (ms) | fwd: 94700.94 | bwd: 101857.44 | bwd_inner: 101580.94 | bwd_allreduce: 95.10 | step: 46.21

 29%|██▉       | 40/138 [2:16:38<5:44:46, 211.08s/it]
                                                     
{'loss': 0.0106, 'grad_norm': 0.0016758110141381621, 'learning_rate': 0.00016772815716257412, 'epoch': 0.29}

 29%|██▉       | 40/138 [2:16:38<5:44:46, 211.08s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.87 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 97914.95 | bwd_microstep: 102918.75 | bwd_inner_microstep: 102687.83 | bwd_allreduce_microstep: 26.91 | step_microstep: 25.71
[Rank 0] time (ms) | fwd: 97917.94 | bwd: 102854.66 | bwd_inner: 102624.34 | bwd_allreduce: 25.31 | step: 25.72

 30%|██▉       | 41/138 [2:20:13<5:43:07, 212.24s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 98296.26 | bwd_microstep: 103034.62 | bwd_inner_microstep: 102571.99 | bwd_allreduce_microstep: 34.72 | step_microstep: 18.51
[Rank 0] time (ms) | fwd: 98294.05 | bwd: 102990.21 | bwd_inner: 102805.94 | bwd_allreduce: 32.64 | step: 18.51

 30%|███       | 42/138 [2:23:45<5:39:43, 212.32s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3539
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 95947.78 | bwd_microstep: 102720.69 | bwd_inner_microstep: 102516.19 | bwd_allreduce_microstep: 17.02 | step_microstep: 19.23
[Rank 0] time (ms) | fwd: 95944.93 | bwd: 102662.69 | bwd_inner: 102468.90 | bwd_allreduce: 17.03 | step: 19.25

 31%|███       | 43/138 [2:27:18<5:36:14, 212.36s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
[Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 1.64 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94474.02 | bwd_microstep: 101381.18 | bwd_inner_microstep: 101009.70 | bwd_allreduce_microstep: 91.23 | step_microstep: 21.20
[Rank 0] time (ms) | fwd: 94469.13 | bwd: 101352.86 | bwd_inner: 101007.78 | bwd_allreduce: 125.31 | step: 21.22

 32%|███▏      | 44/138 [2:30:46<5:30:46, 211.14s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.73
[Rank 0] time (ms) | fwd_microstep: 96563.82 | bwd_microstep: 101327.42 | bwd_inner_microstep: 101115.18 | bwd_allreduce_microstep: 33.07 | step_microstep: 17.81
[Rank 0] time (ms) | fwd: 96556.80 | bwd: 101278.65 | bwd_inner: 101086.93 | bwd_allreduce: 37.26 | step: 17.85

 33%|███▎      | 45/138 [2:34:17<5:27:15, 211.14s/it]
                                                     
{'loss': 0.0123, 'grad_norm': 0.002686451654881239, 'learning_rate': 0.00015858726354602248, 'epoch': 0.33}

 33%|███▎      | 45/138 [2:34:17<5:27:15, 211.14s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1983
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 97267.74 | bwd_microstep: 102225.44 | bwd_inner_microstep: 101931.89 | bwd_allreduce_microstep: 28.85 | step_microstep: 17.99
[Rank 0] time (ms) | fwd: 97259.41 | bwd: 102117.09 | bwd_inner: 101887.36 | bwd_allreduce: 26.96 | step: 18.00

 33%|███▎      | 46/138 [2:37:51<5:25:04, 212.01s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.67 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 95248.44 | bwd_microstep: 102820.63 | bwd_inner_microstep: 102614.04 | bwd_allreduce_microstep: 55.23 | step_microstep: 22.45
[Rank 0] time (ms) | fwd: 95241.71 | bwd: 102787.48 | bwd_inner: 102596.99 | bwd_allreduce: 55.59 | step: 22.45

 34%|███▍      | 47/138 [2:41:20<5:20:14, 211.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 99373.93 | bwd_microstep: 100126.51 | bwd_inner_microstep: 99820.93 | bwd_allreduce_microstep: 68.13 | step_microstep: 17.10
[Rank 0] time (ms) | fwd: 99371.58 | bwd: 100079.28 | bwd_inner: 99799.20 | bwd_allreduce: 63.00 | step: 17.10

 35%|███▍      | 48/138 [2:44:55<5:18:26, 212.30s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 978
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.51 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 96567.63 | bwd_microstep: 97560.36 | bwd_inner_microstep: 97286.98 | bwd_allreduce_microstep: 38.87 | step_microstep: 22.30
[Rank 0] time (ms) | fwd: 96562.39 | bwd: 97487.98 | bwd_inner: 97239.86 | bwd_allreduce: 37.59 | step: 22.38

 36%|███▌      | 49/138 [2:48:24<5:13:10, 211.13s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.08 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 96202.20 | bwd_microstep: 100090.53 | bwd_inner_microstep: 99840.76 | bwd_allreduce_microstep: 34.71 | step_microstep: 18.28
[Rank 0] time (ms) | fwd: 96198.70 | bwd: 100032.57 | bwd_inner: 99807.55 | bwd_allreduce: 30.54 | step: 18.30

 36%|███▌      | 50/138 [2:51:55<5:09:51, 211.26s/it]
                                                     
{'loss': 0.0115, 'grad_norm': 0.0017138413386419415, 'learning_rate': 0.00014863009810942815, 'epoch': 0.36}

 36%|███▌      | 50/138 [2:51:55<5:09:51, 211.26s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.57 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 95424.20 | bwd_microstep: 101586.57 | bwd_inner_microstep: 101338.88 | bwd_allreduce_microstep: 81.41 | step_microstep: 53.46
[Rank 0] time (ms) | fwd: 95421.43 | bwd: 101561.07 | bwd_inner: 101328.36 | bwd_allreduce: 84.02 | step: 53.46

 37%|███▋      | 51/138 [2:55:25<5:05:45, 210.87s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1956
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 98145.79 | bwd_microstep: 100268.65 | bwd_inner_microstep: 99829.43 | bwd_allreduce_microstep: 237.17 | step_microstep: 19.66
[Rank 0] time (ms) | fwd: 98139.78 | bwd: 100214.15 | bwd_inner: 99797.69 | bwd_allreduce: 232.53 | step: 19.66

 38%|███▊      | 52/138 [2:58:57<5:02:43, 211.20s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 975
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94576.53 | bwd_microstep: 100756.71 | bwd_inner_microstep: 100427.76 | bwd_allreduce_microstep: 76.44 | step_microstep: 17.37
[Rank 0] time (ms) | fwd: 94574.48 | bwd: 100710.19 | bwd_inner: 100413.21 | bwd_allreduce: 74.71 | step: 17.38

 38%|███▊      | 53/138 [3:02:24<4:57:31, 210.01s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 96165.19 | bwd_microstep: 98475.60 | bwd_inner_microstep: 98209.73 | bwd_allreduce_microstep: 55.44 | step_microstep: 17.17
[Rank 0] time (ms) | fwd: 96160.45 | bwd: 98426.82 | bwd_inner: 98189.29 | bwd_allreduce: 53.83 | step: 17.17

 39%|███▉      | 54/138 [3:05:54<4:54:05, 210.07s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.34 | optimizer_step: 0.95
[Rank 0] time (ms) | fwd_microstep: 94502.74 | bwd_microstep: 102637.79 | bwd_inner_microstep: 102422.39 | bwd_allreduce_microstep: 49.29 | step_microstep: 43.20
[Rank 0] time (ms) | fwd: 94498.29 | bwd: 102583.14 | bwd_inner: 102390.40 | bwd_allreduce: 49.63 | step: 43.24

 40%|███▉      | 55/138 [3:09:25<4:50:41, 210.13s/it]
                                                     
{'loss': 0.0096, 'grad_norm': 0.003232393180951476, 'learning_rate': 0.000137995389871036, 'epoch': 0.4}

 40%|███▉      | 55/138 [3:09:25<4:50:41, 210.13s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 972
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 96431.38 | bwd_microstep: 99741.50 | bwd_inner_microstep: 99484.66 | bwd_allreduce_microstep: 44.31 | step_microstep: 18.93
[Rank 0] time (ms) | fwd: 96429.07 | bwd: 99671.15 | bwd_inner: 99456.41 | bwd_allreduce: 44.47 | step: 18.93

 41%|████      | 56/138 [3:12:55<4:47:16, 210.20s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 95224.28 | bwd_microstep: 102064.73 | bwd_inner_microstep: 101814.76 | bwd_allreduce_microstep: 31.74 | step_microstep: 18.05
[Rank 0] time (ms) | fwd: 95218.74 | bwd: 101987.34 | bwd_inner: 101768.12 | bwd_allreduce: 32.12 | step: 18.06

 41%|████▏     | 57/138 [3:16:25<4:43:30, 210.01s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 94248.41 | bwd_microstep: 101318.06 | bwd_inner_microstep: 101080.60 | bwd_allreduce_microstep: 42.09 | step_microstep: 18.66
[Rank 0] time (ms) | fwd: 94242.91 | bwd: 101264.39 | bwd_inner: 101045.11 | bwd_allreduce: 33.85 | step: 18.67

 42%|████▏     | 58/138 [3:19:53<4:39:24, 209.56s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 95363.96 | bwd_microstep: 102042.00 | bwd_inner_microstep: 101736.74 | bwd_allreduce_microstep: 53.64 | step_microstep: 17.33
[Rank 0] time (ms) | fwd: 95358.40 | bwd: 101978.36 | bwd_inner: 101697.39 | bwd_allreduce: 35.60 | step: 17.36

 43%|████▎     | 59/138 [3:23:25<4:36:39, 210.12s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 979
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1911
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 96243.65 | bwd_microstep: 97942.62 | bwd_inner_microstep: 97637.74 | bwd_allreduce_microstep: 32.35 | step_microstep: 17.63
[Rank 0] time (ms) | fwd: 96241.09 | bwd: 97878.16 | bwd_inner: 97589.92 | bwd_allreduce: 30.88 | step: 17.64

 43%|████▎     | 60/138 [3:26:53<4:32:38, 209.72s/it]
                                                     
{'loss': 0.014, 'grad_norm': 0.0028239295352250338, 'learning_rate': 0.0001268313077693485, 'epoch': 0.43}

 43%|████▎     | 60/138 [3:26:53<4:32:38, 209.72s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 96152.64 | bwd_microstep: 99615.74 | bwd_inner_microstep: 99415.42 | bwd_allreduce_microstep: 12.08 | step_microstep: 16.92
[Rank 0] time (ms) | fwd: 96128.34 | bwd: 99546.25 | bwd_inner: 99369.81 | bwd_allreduce: 12.22 | step: 16.93

 44%|████▍     | 61/138 [3:30:24<4:29:34, 210.05s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 98385.97 | bwd_microstep: 100292.96 | bwd_inner_microstep: 99959.24 | bwd_allreduce_microstep: 78.13 | step_microstep: 17.83
[Rank 0] time (ms) | fwd: 98383.90 | bwd: 100242.12 | bwd_inner: 99942.88 | bwd_allreduce: 77.95 | step: 17.86

 45%|████▍     | 62/138 [3:33:58<4:27:17, 211.02s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.73
[Rank 0] time (ms) | fwd_microstep: 96148.15 | bwd_microstep: 100505.69 | bwd_inner_microstep: 100088.43 | bwd_allreduce_microstep: 47.96 | step_microstep: 18.61
[Rank 0] time (ms) | fwd: 96124.77 | bwd: 100459.85 | bwd_inner: 100060.97 | bwd_allreduce: 45.48 | step: 18.62

 46%|████▌     | 63/138 [3:37:28<4:23:40, 210.94s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.08 | optimizer_gradients: 5.90 | optimizer_step: 1.48
[Rank 0] time (ms) | fwd_microstep: 95521.45 | bwd_microstep: 101263.29 | bwd_inner_microstep: 101088.45 | bwd_allreduce_microstep: 64.49 | step_microstep: 376.54
[Rank 0] time (ms) | fwd: 95513.34 | bwd: 101230.96 | bwd_inner: 101068.69 | bwd_allreduce: 63.31 | step: 376.51

 46%|████▋     | 64/138 [3:41:00<4:20:26, 211.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 95996.52 | bwd_microstep: 101987.95 | bwd_inner_microstep: 101841.24 | bwd_allreduce_microstep: 12.26 | step_microstep: 17.67
[Rank 0] time (ms) | fwd: 95994.10 | bwd: 101930.95 | bwd_inner: 101798.14 | bwd_allreduce: 12.07 | step: 17.68

 47%|████▋     | 65/138 [3:44:32<4:17:15, 211.44s/it]
                                                     
{'loss': 0.0113, 'grad_norm': 0.0013771996600553393, 'learning_rate': 0.0001152933962873246, 'epoch': 0.47}

 47%|████▋     | 65/138 [3:44:32<4:17:15, 211.44s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.91 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 96270.56 | bwd_microstep: 101877.23 | bwd_inner_microstep: 101704.33 | bwd_allreduce_microstep: 38.03 | step_microstep: 53.70
[Rank 0] time (ms) | fwd: 96262.87 | bwd: 101822.58 | bwd_inner: 101675.35 | bwd_allreduce: 38.48 | step: 53.72

 48%|████▊     | 66/138 [3:48:05<4:14:05, 211.75s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3536
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.63 | optimizer_step: 1.10
[Rank 0] time (ms) | fwd_microstep: 98346.54 | bwd_microstep: 103681.57 | bwd_inner_microstep: 103472.99 | bwd_allreduce_microstep: 40.36 | step_microstep: 37.61
[Rank 0] time (ms) | fwd: 98337.04 | bwd: 103646.90 | bwd_inner: 103459.94 | bwd_allreduce: 39.34 | step: 37.74

 49%|████▊     | 67/138 [3:51:41<4:12:20, 213.24s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.14 | optimizer_step: 0.71
[Rank 0] time (ms) | fwd_microstep: 98441.95 | bwd_microstep: 102041.50 | bwd_inner_microstep: 101874.26 | bwd_allreduce_microstep: 21.19 | step_microstep: 16.61
[Rank 0] time (ms) | fwd: 98439.58 | bwd: 102000.02 | bwd_inner: 101829.97 | bwd_allreduce: 21.78 | step: 16.62

 49%|████▉     | 68/138 [3:55:15<4:09:01, 213.45s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 97937.65 | bwd_microstep: 101297.16 | bwd_inner_microstep: 101034.32 | bwd_allreduce_microstep: 38.21 | step_microstep: 17.81
[Rank 0] time (ms) | fwd: 97935.23 | bwd: 101239.88 | bwd_inner: 100994.52 | bwd_allreduce: 36.04 | step: 17.82

 50%|█████     | 69/138 [3:58:48<4:05:21, 213.35s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.74 | optimizer_step: 0.95
[Rank 0] time (ms) | fwd_microstep: 96170.83 | bwd_microstep: 101260.04 | bwd_inner_microstep: 101043.63 | bwd_allreduce_microstep: 46.10 | step_microstep: 25.77
[Rank 0] time (ms) | fwd: 96161.49 | bwd: 101215.08 | bwd_inner: 101014.21 | bwd_allreduce: 46.48 | step: 25.77

 51%|█████     | 70/138 [4:02:17<4:00:22, 212.10s/it]
                                                     
{'loss': 0.0106, 'grad_norm': 0.0019330934155732393, 'learning_rate': 0.00010354240831620541, 'epoch': 0.51}

 51%|█████     | 70/138 [4:02:18<4:00:22, 212.10s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 96278.03 | bwd_microstep: 98077.34 | bwd_inner_microstep: 97851.85 | bwd_allreduce_microstep: 40.71 | step_microstep: 17.44
[Rank 0] time (ms) | fwd: 96275.61 | bwd: 98034.16 | bwd_inner: 97825.90 | bwd_allreduce: 46.63 | step: 17.44

 51%|█████▏    | 71/138 [4:05:48<3:56:10, 211.50s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 98705.22 | bwd_microstep: 99805.69 | bwd_inner_microstep: 99621.31 | bwd_allreduce_microstep: 56.67 | step_microstep: 19.29
[Rank 0] time (ms) | fwd: 98699.03 | bwd: 99792.95 | bwd_inner: 99611.86 | bwd_allreduce: 55.66 | step: 19.29

 52%|█████▏    | 72/138 [4:09:22<3:53:40, 212.43s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.72
[Rank 0] time (ms) | fwd_microstep: 96258.94 | bwd_microstep: 100119.32 | bwd_inner_microstep: 99859.40 | bwd_allreduce_microstep: 38.91 | step_microstep: 17.00
[Rank 0] time (ms) | fwd: 96256.52 | bwd: 100077.63 | bwd_inner: 99819.65 | bwd_allreduce: 38.16 | step: 17.01

 53%|█████▎    | 73/138 [4:12:53<3:49:28, 211.83s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3530
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.59 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 97263.94 | bwd_microstep: 101855.60 | bwd_inner_microstep: 101641.04 | bwd_allreduce_microstep: 39.19 | step_microstep: 23.96
[Rank 0] time (ms) | fwd: 97258.25 | bwd: 101807.34 | bwd_inner: 101608.60 | bwd_allreduce: 37.73 | step: 23.95

 54%|█████▎    | 74/138 [4:16:26<3:46:35, 212.43s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 97120.74 | bwd_microstep: 100572.77 | bwd_inner_microstep: 100402.06 | bwd_allreduce_microstep: 24.32 | step_microstep: 18.03
[Rank 0] time (ms) | fwd: 97117.96 | bwd: 100522.55 | bwd_inner: 100363.33 | bwd_allreduce: 24.38 | step: 18.03

 54%|█████▍    | 75/138 [4:19:58<3:42:40, 212.08s/it]
                                                     
{'loss': 0.0094, 'grad_norm': 0.0029724629130214453, 'learning_rate': 9.174206545276677e-05, 'epoch': 0.54}

 54%|█████▍    | 75/138 [4:19:58<3:42:40, 212.08s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 97195.78 | bwd_microstep: 100994.16 | bwd_inner_microstep: 100837.83 | bwd_allreduce_microstep: 28.51 | step_microstep: 17.27
[Rank 0] time (ms) | fwd: 97193.61 | bwd: 100959.78 | bwd_inner: 100819.28 | bwd_allreduce: 27.36 | step: 17.28

 55%|█████▌    | 76/138 [4:23:29<3:38:54, 211.84s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.04 | optimizer_gradients: 2.27 | optimizer_step: 1.16
[Rank 0] time (ms) | fwd_microstep: 93753.10 | bwd_microstep: 99517.66 | bwd_inner_microstep: 99253.70 | bwd_allreduce_microstep: 36.85 | step_microstep: 29.83
[Rank 0] time (ms) | fwd: 93750.95 | bwd: 99456.46 | bwd_inner: 99209.58 | bwd_allreduce: 39.48 | step: 29.89

 56%|█████▌    | 77/138 [4:26:57<3:34:03, 210.55s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 96634.08 | bwd_microstep: 103067.63 | bwd_inner_microstep: 102822.71 | bwd_allreduce_microstep: 36.78 | step_microstep: 17.69
[Rank 0] time (ms) | fwd: 96631.29 | bwd: 103036.49 | bwd_inner: 102793.21 | bwd_allreduce: 37.73 | step: 17.70

 57%|█████▋    | 78/138 [4:30:29<3:31:15, 211.26s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 95207.43 | bwd_microstep: 100975.95 | bwd_inner_microstep: 100707.16 | bwd_allreduce_microstep: 50.01 | step_microstep: 18.02
[Rank 0] time (ms) | fwd: 95204.84 | bwd: 100924.23 | bwd_inner: 100675.78 | bwd_allreduce: 55.79 | step: 18.03

 57%|█████▋    | 79/138 [4:33:58<3:26:54, 210.41s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 96141.11 | bwd_microstep: 101379.46 | bwd_inner_microstep: 101081.61 | bwd_allreduce_microstep: 75.60 | step_microstep: 17.07
[Rank 0] time (ms) | fwd: 96138.90 | bwd: 101335.58 | bwd_inner: 101072.97 | bwd_allreduce: 81.65 | step: 17.08

 58%|█████▊    | 80/138 [4:37:28<3:23:12, 210.21s/it]
                                                     
{'loss': 0.0108, 'grad_norm': 0.0018539836164563894, 'learning_rate': 8.005677693484077e-05, 'epoch': 0.58}

 58%|█████▊    | 80/138 [4:37:28<3:23:12, 210.21s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.72
[Rank 0] time (ms) | fwd_microstep: 95399.78 | bwd_microstep: 102338.57 | bwd_inner_microstep: 102101.96 | bwd_allreduce_microstep: 18.37 | step_microstep: 32.99
[Rank 0] time (ms) | fwd: 95395.83 | bwd: 102233.59 | bwd_inner: 102054.14 | bwd_allreduce: 18.37 | step: 33.01

 59%|█████▊    | 81/138 [4:40:56<3:19:15, 209.74s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 95123.24 | bwd_microstep: 99583.07 | bwd_inner_microstep: 99351.93 | bwd_allreduce_microstep: 34.17 | step_microstep: 18.60
[Rank 0] time (ms) | fwd: 95120.61 | bwd: 99530.75 | bwd_inner: 99314.55 | bwd_allreduce: 34.50 | step: 18.61

 59%|█████▉    | 82/138 [4:44:25<3:15:36, 209.59s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 5, images per sample: 5.0, dynamic token length: 1481
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 96626.89 | bwd_microstep: 102466.60 | bwd_inner_microstep: 102245.01 | bwd_allreduce_microstep: 26.86 | step_microstep: 18.65
[Rank 0] time (ms) | fwd: 96618.94 | bwd: 102412.65 | bwd_inner: 102215.32 | bwd_allreduce: 28.22 | step: 18.65

 60%|██████    | 83/138 [4:47:58<3:12:52, 210.42s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 97127.94 | bwd_microstep: 99259.16 | bwd_inner_microstep: 99094.36 | bwd_allreduce_microstep: 11.93 | step_microstep: 17.41
[Rank 0] time (ms) | fwd: 97112.29 | bwd: 99202.65 | bwd_inner: 99055.64 | bwd_allreduce: 12.11 | step: 17.42

 61%|██████    | 84/138 [4:51:31<3:10:06, 211.22s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.79 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 100650.16 | bwd_microstep: 100029.22 | bwd_inner_microstep: 99753.78 | bwd_allreduce_microstep: 35.43 | step_microstep: 31.95
[Rank 0] time (ms) | fwd: 100647.50 | bwd: 99962.28 | bwd_inner: 99706.87 | bwd_allreduce: 37.50 | step: 31.98

 62%|██████▏   | 85/138 [4:55:07<3:07:56, 212.77s/it]
                                                     
{'loss': 0.008, 'grad_norm': 0.0012941986788064241, 'learning_rate': 6.864934899622191e-05, 'epoch': 0.62}

 62%|██████▏   | 85/138 [4:55:07<3:07:56, 212.77s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3530
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.03 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 95790.82 | bwd_microstep: 102375.40 | bwd_inner_microstep: 102158.88 | bwd_allreduce_microstep: 12.48 | step_microstep: 36.32
[Rank 0] time (ms) | fwd: 95785.31 | bwd: 102304.08 | bwd_inner: 102104.88 | bwd_allreduce: 12.97 | step: 36.34

 62%|██████▏   | 86/138 [4:58:38<3:03:54, 212.20s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.05 | optimizer_gradients: 3.79 | optimizer_step: 1.06
[Rank 0] time (ms) | fwd_microstep: 96692.58 | bwd_microstep: 99842.85 | bwd_inner_microstep: 99554.80 | bwd_allreduce_microstep: 72.36 | step_microstep: 61.62
[Rank 0] time (ms) | fwd: 96689.93 | bwd: 99786.27 | bwd_inner: 99517.03 | bwd_allreduce: 44.09 | step: 61.66

 63%|██████▎   | 87/138 [5:02:10<3:00:13, 212.03s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 94189.43 | bwd_microstep: 99751.18 | bwd_inner_microstep: 99470.66 | bwd_allreduce_microstep: 54.64 | step_microstep: 20.61
[Rank 0] time (ms) | fwd: 94183.82 | bwd: 99691.31 | bwd_inner: 99383.47 | bwd_allreduce: 52.07 | step: 20.61

 64%|██████▍   | 88/138 [5:05:38<2:55:38, 210.76s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 4.30 | optimizer_step: 1.01
[Rank 0] time (ms) | fwd_microstep: 95436.64 | bwd_microstep: 102981.48 | bwd_inner_microstep: 102683.09 | bwd_allreduce_microstep: 106.60 | step_microstep: 49.05
[Rank 0] time (ms) | fwd: 95434.56 | bwd: 102955.70 | bwd_inner: 102662.34 | bwd_allreduce: 106.85 | step: 49.03

 64%|██████▍   | 89/138 [5:09:07<2:51:47, 210.35s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.17 | optimizer_gradients: 5.73 | optimizer_step: 1.50
[Rank 0] time (ms) | fwd_microstep: 96832.85 | bwd_microstep: 101175.95 | bwd_inner_microstep: 100953.43 | bwd_allreduce_microstep: 50.56 | step_microstep: 115.07
[Rank 0] time (ms) | fwd: 96823.93 | bwd: 101127.86 | bwd_inner: 100913.11 | bwd_allreduce: 48.87 | step: 138.31

 65%|██████▌   | 90/138 [5:12:39<2:48:43, 210.90s/it]
                                                     
{'loss': 0.009, 'grad_norm': 0.0015225649112835526, 'learning_rate': 5.767871655555751e-05, 'epoch': 0.65}

 65%|██████▌   | 90/138 [5:12:39<2:48:43, 210.90s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.71 | optimizer_step: 1.21
[Rank 0] time (ms) | fwd_microstep: 96815.18 | bwd_microstep: 97784.45 | bwd_inner_microstep: 97444.79 | bwd_allreduce_microstep: 43.55 | step_microstep: 30.09
[Rank 0] time (ms) | fwd: 96808.44 | bwd: 97695.60 | bwd_inner: 97378.76 | bwd_allreduce: 42.60 | step: 30.14

 66%|██████▌   | 91/138 [5:16:11<2:45:25, 211.18s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.35 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 98229.93 | bwd_microstep: 102834.86 | bwd_inner_microstep: 102557.69 | bwd_allreduce_microstep: 60.25 | step_microstep: 30.42
[Rank 0] time (ms) | fwd: 98223.68 | bwd: 102797.22 | bwd_inner: 102531.80 | bwd_allreduce: 63.11 | step: 30.47

 67%|██████▋   | 92/138 [5:19:45<2:42:26, 211.89s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3527
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 97044.38 | bwd_microstep: 104700.60 | bwd_inner_microstep: 104461.10 | bwd_allreduce_microstep: 60.47 | step_microstep: 18.77
[Rank 0] time (ms) | fwd: 97042.51 | bwd: 104670.24 | bwd_inner: 104439.88 | bwd_allreduce: 65.65 | step: 18.78

 67%|██████▋   | 93/138 [5:23:17<2:39:08, 212.18s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3539
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.73
[Rank 0] time (ms) | fwd_microstep: 96640.33 | bwd_microstep: 102745.55 | bwd_inner_microstep: 102459.43 | bwd_allreduce_microstep: 80.54 | step_microstep: 17.38
[Rank 0] time (ms) | fwd: 96637.96 | bwd: 102711.65 | bwd_inner: 102454.13 | bwd_allreduce: 76.23 | step: 17.39

 68%|██████▊   | 94/138 [5:26:49<2:35:31, 212.07s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 973
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 94934.99 | bwd_microstep: 101776.93 | bwd_inner_microstep: 101490.97 | bwd_allreduce_microstep: 45.81 | step_microstep: 22.25
[Rank 0] time (ms) | fwd: 94932.72 | bwd: 101711.65 | bwd_inner: 101447.75 | bwd_allreduce: 47.12 | step: 22.24

 69%|██████▉   | 95/138 [5:30:18<2:31:17, 211.10s/it]
                                                     
{'loss': 0.0098, 'grad_norm': 0.0012339482782408595, 'learning_rate': 4.729772884265212e-05, 'epoch': 0.69}

 69%|██████▉   | 95/138 [5:30:18<2:31:17, 211.10s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 96274.44 | bwd_microstep: 102672.24 | bwd_inner_microstep: 102476.22 | bwd_allreduce_microstep: 26.05 | step_microstep: 17.85
[Rank 0] time (ms) | fwd: 96272.14 | bwd: 102626.15 | bwd_inner: 102434.26 | bwd_allreduce: 28.91 | step: 17.86

 70%|██████▉   | 96/138 [5:33:48<2:27:37, 210.89s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.55 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 93882.62 | bwd_microstep: 103071.59 | bwd_inner_microstep: 102884.38 | bwd_allreduce_microstep: 33.37 | step_microstep: 18.67
[Rank 0] time (ms) | fwd: 93880.68 | bwd: 103037.65 | bwd_inner: 102864.08 | bwd_allreduce: 34.85 | step: 18.67

 70%|███████   | 97/138 [5:37:17<2:23:36, 210.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 94778.92 | bwd_microstep: 103052.62 | bwd_inner_microstep: 102830.92 | bwd_allreduce_microstep: 52.90 | step_microstep: 18.64
[Rank 0] time (ms) | fwd: 94776.94 | bwd: 103027.80 | bwd_inner: 102819.64 | bwd_allreduce: 60.81 | step: 18.64

 71%|███████   | 98/138 [5:40:45<2:19:45, 209.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.16 | optimizer_gradients: 3.52 | optimizer_step: 1.56
[Rank 0] time (ms) | fwd_microstep: 94924.92 | bwd_microstep: 105370.19 | bwd_inner_microstep: 105266.83 | bwd_allreduce_microstep: 31.36 | step_microstep: 73.72
[Rank 0] time (ms) | fwd: 94923.26 | bwd: 105353.77 | bwd_inner: 105254.36 | bwd_allreduce: 33.10 | step: 73.76

 72%|███████▏  | 99/138 [5:44:16<2:16:25, 209.88s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.34 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 99237.06 | bwd_microstep: 101438.38 | bwd_inner_microstep: 101213.11 | bwd_allreduce_microstep: 29.31 | step_microstep: 21.71
[Rank 0] time (ms) | fwd: 99230.74 | bwd: 101378.94 | bwd_inner: 101166.86 | bwd_allreduce: 30.69 | step: 21.72

 72%|███████▏  | 100/138 [5:47:49<2:13:31, 210.82s/it]
                                                      
{'loss': 0.0071, 'grad_norm': 0.0009091832325793803, 'learning_rate': 3.7651019814126654e-05, 'epoch': 0.72}

 72%|███████▏  | 100/138 [5:47:49<2:13:31, 210.82s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 6.41 | optimizer_gradients: 36.73 | optimizer_step: 26.31
[Rank 0] time (ms) | fwd_microstep: 99283.72 | bwd_microstep: 98366.38 | bwd_inner_microstep: 98123.54 | bwd_allreduce_microstep: 35.06 | step_microstep: 297.92
[Rank 0] time (ms) | fwd: 99280.68 | bwd: 98277.43 | bwd_inner: 98049.61 | bwd_allreduce: 35.18 | step: 297.98

 73%|███████▎  | 101/138 [5:51:24<2:10:48, 212.11s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3540
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 98166.79 | bwd_microstep: 100507.74 | bwd_inner_microstep: 100229.96 | bwd_allreduce_microstep: 52.55 | step_microstep: 19.23
[Rank 0] time (ms) | fwd: 98164.52 | bwd: 100444.50 | bwd_inner: 100201.69 | bwd_allreduce: 55.50 | step: 19.25

 74%|███████▍  | 102/138 [5:54:58<2:07:34, 212.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.14 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 94810.11 | bwd_microstep: 101054.88 | bwd_inner_microstep: 100833.17 | bwd_allreduce_microstep: 65.08 | step_microstep: 17.44
[Rank 0] time (ms) | fwd: 94807.46 | bwd: 101032.26 | bwd_inner: 100823.97 | bwd_allreduce: 68.83 | step: 17.44

 75%|███████▍  | 103/138 [5:58:27<2:03:28, 211.66s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.90 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 96425.68 | bwd_microstep: 101811.25 | bwd_inner_microstep: 101597.10 | bwd_allreduce_microstep: 36.64 | step_microstep: 25.44
[Rank 0] time (ms) | fwd: 96423.57 | bwd: 101765.96 | bwd_inner: 101561.24 | bwd_allreduce: 35.81 | step: 25.44

 75%|███████▌  | 104/138 [6:01:58<1:59:46, 211.37s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.06 | optimizer_gradients: 1.68 | optimizer_step: 1.17
[Rank 0] time (ms) | fwd_microstep: 97419.17 | bwd_microstep: 100189.39 | bwd_inner_microstep: 100030.43 | bwd_allreduce_microstep: 12.43 | step_microstep: 26.36
[Rank 0] time (ms) | fwd: 97416.82 | bwd: 100138.97 | bwd_inner: 99992.33 | bwd_allreduce: 12.44 | step: 26.42

 76%|███████▌  | 105/138 [6:05:28<1:55:58, 210.88s/it]
                                                      
{'loss': 0.0085, 'grad_norm': 0.0018332460895180702, 'learning_rate': 2.8872993029040508e-05, 'epoch': 0.76}

 76%|███████▌  | 105/138 [6:05:28<1:55:58, 210.88s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 95041.92 | bwd_microstep: 104026.36 | bwd_inner_microstep: 103775.59 | bwd_allreduce_microstep: 59.70 | step_microstep: 16.99
[Rank 0] time (ms) | fwd: 95033.69 | bwd: 103991.61 | bwd_inner: 103759.81 | bwd_allreduce: 62.09 | step: 17.00

 77%|███████▋  | 106/138 [6:08:57<1:52:09, 210.30s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.18 | optimizer_gradients: 4.69 | optimizer_step: 1.54
[Rank 0] time (ms) | fwd_microstep: 97517.85 | bwd_microstep: 99763.88 | bwd_inner_microstep: 99548.92 | bwd_allreduce_microstep: 36.90 | step_microstep: 48.82
[Rank 0] time (ms) | fwd: 97482.07 | bwd: 99721.52 | bwd_inner: 99504.84 | bwd_allreduce: 39.06 | step: 48.89

 78%|███████▊  | 107/138 [6:12:30<1:49:05, 211.14s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 96220.87 | bwd_microstep: 104613.04 | bwd_inner_microstep: 104390.91 | bwd_allreduce_microstep: 76.18 | step_microstep: 18.73
[Rank 0] time (ms) | fwd: 96218.47 | bwd: 104614.52 | bwd_inner: 104400.74 | bwd_allreduce: 76.98 | step: 18.73

 78%|███████▊  | 108/138 [6:16:01<1:45:38, 211.27s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 98286.41 | bwd_microstep: 99282.59 | bwd_inner_microstep: 99058.00 | bwd_allreduce_microstep: 12.52 | step_microstep: 19.62
[Rank 0] time (ms) | fwd: 98280.00 | bwd: 99204.13 | bwd_inner: 98994.62 | bwd_allreduce: 12.54 | step: 19.63

 79%|███████▉  | 109/138 [6:19:35<1:42:25, 211.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.72
[Rank 0] time (ms) | fwd_microstep: 96988.56 | bwd_microstep: 99448.87 | bwd_inner_microstep: 99127.54 | bwd_allreduce_microstep: 56.52 | step_microstep: 17.85
[Rank 0] time (ms) | fwd: 96986.36 | bwd: 99386.62 | bwd_inner: 99083.20 | bwd_allreduce: 62.76 | step: 17.86

 80%|███████▉  | 110/138 [6:23:05<1:38:40, 211.43s/it]
                                                      
{'loss': 0.0088, 'grad_norm': 0.0009983633644878864, 'learning_rate': 2.1085949060360654e-05, 'epoch': 0.8}

 80%|███████▉  | 110/138 [6:23:05<1:38:40, 211.43s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 94719.14 | bwd_microstep: 101461.06 | bwd_inner_microstep: 101190.68 | bwd_allreduce_microstep: 68.86 | step_microstep: 17.50
[Rank 0] time (ms) | fwd: 94710.87 | bwd: 101417.84 | bwd_inner: 101150.07 | bwd_allreduce: 65.59 | step: 17.51

 80%|████████  | 111/138 [6:26:35<1:34:55, 210.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.65 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 97851.47 | bwd_microstep: 102980.55 | bwd_inner_microstep: 102846.15 | bwd_allreduce_microstep: 20.21 | step_microstep: 25.65
[Rank 0] time (ms) | fwd: 97835.32 | bwd: 102944.04 | bwd_inner: 102802.33 | bwd_allreduce: 20.40 | step: 25.66

 81%|████████  | 112/138 [6:30:08<1:31:39, 211.51s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 96133.82 | bwd_microstep: 105230.46 | bwd_inner_microstep: 105035.71 | bwd_allreduce_microstep: 65.35 | step_microstep: 18.02
[Rank 0] time (ms) | fwd: 96131.61 | bwd: 105194.86 | bwd_inner: 105016.27 | bwd_allreduce: 67.69 | step: 18.02

 82%|████████▏ | 113/138 [6:33:39<1:28:04, 211.39s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3528
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.73
[Rank 0] time (ms) | fwd_microstep: 96520.73 | bwd_microstep: 103372.75 | bwd_inner_microstep: 103115.18 | bwd_allreduce_microstep: 39.91 | step_microstep: 17.94
[Rank 0] time (ms) | fwd: 96518.42 | bwd: 103339.34 | bwd_inner: 103078.59 | bwd_allreduce: 40.48 | step: 17.97

 83%|████████▎ | 114/138 [6:37:11<1:24:37, 211.56s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3538
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.14 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 96073.06 | bwd_microstep: 102830.69 | bwd_inner_microstep: 102631.03 | bwd_allreduce_microstep: 10.05 | step_microstep: 16.92
[Rank 0] time (ms) | fwd: 96070.89 | bwd: 102769.41 | bwd_inner: 102567.02 | bwd_allreduce: 10.08 | step: 16.93

 83%|████████▎ | 115/138 [6:40:41<1:21:00, 211.31s/it]
                                                      
{'loss': 0.0074, 'grad_norm': 0.001077743829227984, 'learning_rate': 1.439838153227e-05, 'epoch': 0.83}

 83%|████████▎ | 115/138 [6:40:41<1:21:00, 211.31s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3447
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 97353.72 | bwd_microstep: 100773.33 | bwd_inner_microstep: 100596.88 | bwd_allreduce_microstep: 34.28 | step_microstep: 18.20
[Rank 0] time (ms) | fwd: 97351.21 | bwd: 100734.96 | bwd_inner: 100570.34 | bwd_allreduce: 31.82 | step: 18.21

 84%|████████▍ | 116/138 [6:44:14<1:17:35, 211.59s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 96217.08 | bwd_microstep: 101482.26 | bwd_inner_microstep: 101309.85 | bwd_allreduce_microstep: 11.28 | step_microstep: 17.09
[Rank 0] time (ms) | fwd: 96218.77 | bwd: 101419.09 | bwd_inner: 101266.54 | bwd_allreduce: 11.32 | step: 17.12

 85%|████████▍ | 117/138 [6:47:44<1:13:53, 211.13s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 95957.57 | bwd_microstep: 98371.45 | bwd_inner_microstep: 97981.99 | bwd_allreduce_microstep: 67.48 | step_microstep: 18.33
[Rank 0] time (ms) | fwd: 95954.83 | bwd: 98307.74 | bwd_inner: 97949.01 | bwd_allreduce: 70.01 | step: 18.34

 86%|████████▌ | 118/138 [6:51:14<1:10:18, 210.92s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 94984.53 | bwd_microstep: 99016.71 | bwd_inner_microstep: 98765.70 | bwd_allreduce_microstep: 11.22 | step_microstep: 18.04
[Rank 0] time (ms) | fwd: 94973.61 | bwd: 98934.01 | bwd_inner: 98702.31 | bwd_allreduce: 11.49 | step: 18.05

 86%|████████▌ | 119/138 [6:54:46<1:06:51, 211.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.42 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 97487.08 | bwd_microstep: 99868.21 | bwd_inner_microstep: 99525.12 | bwd_allreduce_microstep: 70.65 | step_microstep: 54.16
[Rank 0] time (ms) | fwd: 97473.66 | bwd: 99808.69 | bwd_inner: 99478.82 | bwd_allreduce: 72.55 | step: 54.21

 87%|████████▋ | 120/138 [6:58:20<1:03:35, 211.98s/it]
                                                      
{'loss': 0.0111, 'grad_norm': 0.0009710977319628, 'learning_rate': 8.903465523913957e-06, 'epoch': 0.87}

 87%|████████▋ | 120/138 [6:58:20<1:03:35, 211.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3536
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.06 | optimizer_gradients: 4.84 | optimizer_step: 1.32
[Rank 0] time (ms) | fwd_microstep: 96214.82 | bwd_microstep: 100944.55 | bwd_inner_microstep: 100680.72 | bwd_allreduce_microstep: 57.46 | step_microstep: 308.48
[Rank 0] time (ms) | fwd: 96212.47 | bwd: 100877.44 | bwd_inner: 100636.47 | bwd_allreduce: 56.71 | step: 308.49

 88%|████████▊ | 121/138 [7:01:51<59:58, 211.68s/it]  dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 96397.43 | bwd_microstep: 100960.90 | bwd_inner_microstep: 100692.89 | bwd_allreduce_microstep: 54.71 | step_microstep: 18.55
[Rank 0] time (ms) | fwd: 96395.13 | bwd: 100927.34 | bwd_inner: 100667.17 | bwd_allreduce: 58.41 | step: 18.54

 88%|████████▊ | 122/138 [7:05:22<56:23, 211.50s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 96572.32 | bwd_microstep: 103074.62 | bwd_inner_microstep: 102850.84 | bwd_allreduce_microstep: 51.50 | step_microstep: 17.53
[Rank 0] time (ms) | fwd: 96570.06 | bwd: 103039.84 | bwd_inner: 102826.47 | bwd_allreduce: 51.80 | step: 17.54

 89%|████████▉ | 123/138 [7:08:54<52:53, 211.60s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.21 | optimizer_gradients: 4.91 | optimizer_step: 1.72
[Rank 0] time (ms) | fwd_microstep: 97816.66 | bwd_microstep: 102700.12 | bwd_inner_microstep: 102553.37 | bwd_allreduce_microstep: 36.00 | step_microstep: 72.55
[Rank 0] time (ms) | fwd: 97810.67 | bwd: 102665.82 | bwd_inner: 102528.31 | bwd_allreduce: 33.62 | step: 72.54

 90%|████████▉ | 124/138 [7:12:29<49:37, 212.65s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 97710.52 | bwd_microstep: 103348.72 | bwd_inner_microstep: 103054.62 | bwd_allreduce_microstep: 38.39 | step_microstep: 18.53
[Rank 0] time (ms) | fwd: 97707.93 | bwd: 103319.94 | bwd_inner: 103046.12 | bwd_allreduce: 37.91 | step: 18.54

 91%|█████████ | 125/138 [7:16:03<46:09, 213.07s/it]
                                                    
{'loss': 0.0104, 'grad_norm': 0.001384108909405768, 'learning_rate': 4.6777594000230855e-06, 'epoch': 0.9}

 91%|█████████ | 125/138 [7:16:03<46:09, 213.07s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 98512.76 | bwd_microstep: 101769.56 | bwd_inner_microstep: 101512.96 | bwd_allreduce_microstep: 22.90 | step_microstep: 18.36
[Rank 0] time (ms) | fwd: 98510.33 | bwd: 101699.07 | bwd_inner: 101452.62 | bwd_allreduce: 23.82 | step: 18.36

 91%|█████████▏| 126/138 [7:19:35<42:33, 212.77s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3526
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.10 | optimizer_step: 0.71
[Rank 0] time (ms) | fwd_microstep: 96077.84 | bwd_microstep: 102332.45 | bwd_inner_microstep: 102182.85 | bwd_allreduce_microstep: 31.33 | step_microstep: 16.44
[Rank 0] time (ms) | fwd: 96071.65 | bwd: 102312.20 | bwd_inner: 102164.68 | bwd_allreduce: 43.14 | step: 16.46

 92%|█████████▏| 127/138 [7:23:06<38:55, 212.31s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 96778.95 | bwd_microstep: 101167.79 | bwd_inner_microstep: 100957.67 | bwd_allreduce_microstep: 11.69 | step_microstep: 17.96
[Rank 0] time (ms) | fwd: 96769.17 | bwd: 101090.54 | bwd_inner: 100904.70 | bwd_allreduce: 11.81 | step: 17.95

 93%|█████████▎| 128/138 [7:26:38<35:22, 212.21s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.13 | optimizer_step: 0.74
[Rank 0] time (ms) | fwd_microstep: 96970.11 | bwd_microstep: 100452.15 | bwd_inner_microstep: 100207.03 | bwd_allreduce_microstep: 38.58 | step_microstep: 16.79
[Rank 0] time (ms) | fwd: 96976.09 | bwd: 100386.33 | bwd_inner: 100168.05 | bwd_allreduce: 37.73 | step: 16.79

 93%|█████████▎| 129/138 [7:30:11<31:51, 212.44s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93383.63 | bwd_microstep: 103528.21 | bwd_inner_microstep: 103371.72 | bwd_allreduce_microstep: 21.41 | step_microstep: 17.78
[Rank 0] time (ms) | fwd: 93381.80 | bwd: 103478.92 | bwd_inner: 103345.10 | bwd_allreduce: 21.42 | step: 17.76

 94%|█████████▍| 130/138 [7:33:37<28:04, 210.53s/it]
                                                    
{'loss': 0.0092, 'grad_norm': 0.0009462545858696103, 'learning_rate': 1.7801381552624563e-06, 'epoch': 0.94}

 94%|█████████▍| 130/138 [7:33:37<28:04, 210.53s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93807.25 | bwd_microstep: 99951.52 | bwd_inner_microstep: 99733.26 | bwd_allreduce_microstep: 52.45 | step_microstep: 18.31
[Rank 0] time (ms) | fwd: 93804.86 | bwd: 99924.33 | bwd_inner: 99719.99 | bwd_allreduce: 48.56 | step: 18.33

 95%|█████████▍| 131/138 [7:37:02<24:21, 208.74s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.55 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93789.39 | bwd_microstep: 98952.20 | bwd_inner_microstep: 98655.26 | bwd_allreduce_microstep: 127.52 | step_microstep: 23.63
[Rank 0] time (ms) | fwd: 93787.02 | bwd: 98897.53 | bwd_inner: 98636.29 | bwd_allreduce: 124.01 | step: 23.63

 96%|█████████▌| 132/138 [7:40:25<20:42, 207.09s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 972
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93227.45 | bwd_microstep: 101912.31 | bwd_inner_microstep: 101758.71 | bwd_allreduce_microstep: 37.67 | step_microstep: 18.10
[Rank 0] time (ms) | fwd: 93225.90 | bwd: 101893.30 | bwd_inner: 101744.37 | bwd_allreduce: 36.18 | step: 18.10

 96%|█████████▋| 133/138 [7:43:46<17:06, 205.22s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 978
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 92410.56 | bwd_microstep: 103905.33 | bwd_inner_microstep: 103709.83 | bwd_allreduce_microstep: 48.13 | step_microstep: 18.44
[Rank 0] time (ms) | fwd: 92408.98 | bwd: 103880.07 | bwd_inner: 103708.09 | bwd_allreduce: 49.79 | step: 18.45

 97%|█████████▋| 134/138 [7:47:08<13:37, 204.45s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.50 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 94866.67 | bwd_microstep: 100445.54 | bwd_inner_microstep: 100130.76 | bwd_allreduce_microstep: 106.93 | step_microstep: 38.39
[Rank 0] time (ms) | fwd: 94864.45 | bwd: 100420.27 | bwd_inner: 100128.21 | bwd_allreduce: 109.33 | step: 38.38

 98%|█████████▊| 135/138 [7:50:34<10:14, 204.87s/it]
                                                    
{'loss': 0.0088, 'grad_norm': 0.0013169609010219574, 'learning_rate': 2.509731335744281e-07, 'epoch': 0.98}

 98%|█████████▊| 135/138 [7:50:34<10:14, 204.87s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 93974.38 | bwd_microstep: 100295.13 | bwd_inner_microstep: 100029.64 | bwd_allreduce_microstep: 46.36 | step_microstep: 17.29
[Rank 0] time (ms) | fwd: 93972.45 | bwd: 100227.95 | bwd_inner: 99989.86 | bwd_allreduce: 51.13 | step: 17.29

 99%|█████████▊| 136/138 [7:53:58<06:49, 204.67s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 979
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 93118.78 | bwd_microstep: 99325.10 | bwd_inner_microstep: 99044.85 | bwd_allreduce_microstep: 69.41 | step_microstep: 17.63
[Rank 0] time (ms) | fwd: 93117.16 | bwd: 99296.21 | bwd_inner: 99078.64 | bwd_allreduce: 66.95 | step: 17.64

 99%|█████████▉| 137/138 [7:57:19<03:23, 203.53s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.31 | optimizer_step: 0.94
[Rank 0] time (ms) | fwd_microstep: 93001.73 | bwd_microstep: 99578.86 | bwd_inner_microstep: 99236.12 | bwd_allreduce_microstep: 107.78 | step_microstep: 34.26
[Rank 0] time (ms) | fwd: 93006.02 | bwd: 99564.41 | bwd_inner: 99261.67 | bwd_allreduce: 104.90 | step: 34.31

100%|██████████| 138/138 [8:00:41<00:00, 202.86s/it]petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[INFO|trainer.py:2329] 2026-07-06 17:34:37,193 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



                                                    
{'train_runtime': 28844.8665, 'train_samples_per_second': 0.307, 'train_steps_per_second': 0.005, 'train_loss': 0.02029224556695292, 'epoch': 1.0}

100%|██████████| 138/138 [8:00:42<00:00, 202.86s/it]
100%|██████████| 138/138 [8:00:42<00:00, 209.01s/it]
[INFO|trainer.py:3410] 2026-07-06 17:34:39,243 >> Saving model checkpoint to /root/ai-project/Vintern/vlm_training
[INFO|configuration_utils.py:472] 2026-07-06 17:34:39,252 >> Configuration saved in /root/ai-project/Vintern/vlm_training/config.json
[INFO|configuration_utils.py:731] 2026-07-06 17:34:39,253 >> Configuration saved in /root/ai-project/Vintern/vlm_training/generation_config.json
[INFO|modeling_utils.py:2618] 2026-07-06 17:34:47,045 >> Model weights saved in /root/ai-project/Vintern/vlm_training/model.safetensors
[INFO|tokenization_utils_base.py:2513] 2026-07-06 17:34:47,047 >> tokenizer config file saved in /root/ai-project/Vintern/vlm_training/tokenizer_config.json
[INFO|tokenization_utils_base.py:2522] 2026-07-06 17:34:47,048 >> Special tokens file saved in /root/ai-project/Vintern/vlm_training/special_tokens_map.json
[INFO|tokenization_utils_base.py:2573] 2026-07-06 17:34:47,048 >> added tokens file saved in /root/ai-project/Vintern/vlm_training/added_tokens.json
***** train metrics *****
  epoch                    =         0.9988
  total_flos               = 169354287872GF
  train_loss               =         0.0203
  train_runtime            =     8:00:44.86
  train_samples            =           8843
  train_samples_per_second =          0.307
  train_steps_per_second   =          0.005
wandb: 
wandb: 🚀 View run vintern_lora_card at: https://wandb.ai/hhung26062000/huggingface/runs/ogttutpj
wandb: Find logs at: wandb/run-20260706_093353-ogttutpj/logs
[rank0]:[W706 17:34:50.050944683 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
07/06/2026 17:47:08 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
07/06/2026 17:47:08 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=4,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=zero_stage1_config.json,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_steps=None,
eval_strategy=no,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=64,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=True,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=every_save,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0002,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/root/ai-project/Vintern/vlm_training/runs/Jul06_17-47-08_data-ai-gpu-01,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=5,
logging_strategy=steps,
lr_scheduler_kwargs={},
lr_scheduler_type=cosine,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=2.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
output_dir=/root/ai-project/Vintern/vlm_training,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=vintern_lora_card,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=steps,
save_total_limit=2,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_mps_device=False,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.01,
)
07/06/2026 17:47:08 - INFO - __main__ - Loading Tokenizer: /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5
[INFO|tokenization_utils_base.py:2106] 2026-07-06 17:47:08,945 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 17:47:08,945 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2106] 2026-07-06 17:47:08,945 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 17:47:08,945 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 17:47:08,945 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2106] 2026-07-06 17:47:08,945 >> loading file tokenizer.json
[WARNING|logging.py:314] 2026-07-06 17:47:09,256 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/06/2026 17:47:09 - INFO - __main__ - Loading InternVLChatModel...
[INFO|configuration_utils.py:731] 2026-07-06 17:47:09,265 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/config.json
[INFO|configuration_utils.py:796] 2026-07-06 17:47:09,266 >> Model config InternVLChatConfig {
  "_commit_hash": null,
  "architectures": [
    "InternVLChatModel"
  ],
  "auto_map": {
    "AutoConfig": "5CD-AI/Vintern-1B-v3_5--configuration_internvl_chat.InternVLChatConfig",
    "AutoModel": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel",
    "AutoModelForCausalLM": "5CD-AI/Vintern-1B-v3_5--modeling_internvl_chat.InternVLChatModel"
  },
  "downsample_ratio": 0.5,
  "dynamic_image_size": true,
  "force_image_size": 448,
  "llm_config": {
    "_name_or_path": "Qwen/Qwen2.5-0.5B-Instruct",
    "add_cross_attention": false,
    "architectures": [
      "Qwen2ForCausalLM"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": 151643,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": 151645,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "silu",
    "hidden_size": 896,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "initializer_range": 0.02,
    "intermediate_size": 4864,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "length_penalty": 1.0,
    "max_length": 20,
    "max_position_embeddings": 32768,
    "max_window_layers": 21,
    "min_length": 0,
    "model_type": "qwen2",
    "no_repeat_ngram_size": 0,
    "num_attention_heads": 14,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_hidden_layers": 24,
    "num_key_value_heads": 2,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "rms_norm_eps": 1e-06,
    "rope_theta": 1000000.0,
    "sep_token_id": null,
    "sliding_window": 32768,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": false,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_cache": false,
    "use_sliding_window": false,
    "vocab_size": 151674
  },
  "max_dynamic_patch": 4,
  "min_dynamic_patch": 1,
  "model_type": "internvl_chat",
  "pad2square": false,
  "ps_version": "v2",
  "select_layer": -1,
  "template": "Hermes-2",
  "torch_dtype": "float32",
  "transformers_version": null,
  "use_backbone_lora": 0,
  "use_llm_lora": 0,
  "use_thumbnail": true,
  "vision_config": {
    "_name_or_path": "",
    "add_cross_attention": false,
    "architectures": [
      "InternVisionModel"
    ],
    "attention_dropout": 0.0,
    "bad_words_ids": null,
    "begin_suppress_tokens": null,
    "bos_token_id": null,
    "chunk_size_feed_forward": 0,
    "cross_attention_hidden_size": null,
    "decoder_start_token_id": null,
    "diversity_penalty": 0.0,
    "do_sample": false,
    "drop_path_rate": 0.0,
    "dropout": 0.0,
    "early_stopping": false,
    "encoder_no_repeat_ngram_size": 0,
    "eos_token_id": null,
    "exponential_decay_length_penalty": null,
    "finetuning_task": null,
    "forced_bos_token_id": null,
    "forced_eos_token_id": null,
    "hidden_act": "gelu",
    "hidden_size": 1024,
    "id2label": {
      "0": "LABEL_0",
      "1": "LABEL_1"
    },
    "image_size": 448,
    "initializer_factor": 1.0,
    "initializer_range": 0.02,
    "intermediate_size": 4096,
    "is_decoder": false,
    "is_encoder_decoder": false,
    "label2id": {
      "LABEL_0": 0,
      "LABEL_1": 1
    },
    "layer_norm_eps": 1e-06,
    "length_penalty": 1.0,
    "max_length": 20,
    "min_length": 0,
    "model_type": "intern_vit_6b",
    "no_repeat_ngram_size": 0,
    "norm_type": "layer_norm",
    "num_attention_heads": 16,
    "num_beam_groups": 1,
    "num_beams": 1,
    "num_channels": 3,
    "num_hidden_layers": 24,
    "num_return_sequences": 1,
    "output_attentions": false,
    "output_hidden_states": false,
    "output_scores": false,
    "pad_token_id": null,
    "patch_size": 14,
    "prefix": null,
    "problem_type": null,
    "pruned_heads": {},
    "qk_normalization": false,
    "qkv_bias": true,
    "remove_invalid_values": false,
    "repetition_penalty": 1.0,
    "return_dict": true,
    "return_dict_in_generate": false,
    "sep_token_id": null,
    "suppress_tokens": null,
    "task_specific_params": null,
    "temperature": 1.0,
    "tf_legacy_loss": false,
    "tie_encoder_decoder": false,
    "tie_word_embeddings": true,
    "tokenizer_class": null,
    "top_k": 50,
    "top_p": 1.0,
    "torch_dtype": "bfloat16",
    "torchscript": false,
    "transformers_version": "4.41.2",
    "typical_p": 1.0,
    "use_bfloat16": true,
    "use_flash_attn": false
  }
}

07/06/2026 17:47:09 - INFO - __main__ - Using flash_attention_2 for LLaMA
[INFO|modeling_utils.py:3471] 2026-07-06 17:47:09,267 >> loading weights file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/model.safetensors
[INFO|modeling_utils.py:1519] 2026-07-06 17:47:09,290 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:962] 2026-07-06 17:47:09,292 >> Generate config GenerationConfig {}

[INFO|configuration_utils.py:962] 2026-07-06 17:47:09,348 >> Generate config GenerationConfig {
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "use_cache": false
}

[INFO|modeling_utils.py:4280] 2026-07-06 17:47:10,512 >> All model checkpoint weights were used when initializing InternVLChatModel.

[INFO|modeling_utils.py:4288] 2026-07-06 17:47:10,513 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5.
If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training.
[INFO|configuration_utils.py:915] 2026-07-06 17:47:10,517 >> loading configuration file /root/ai-project/Vintern/pretrained/Vintern-1B-v3_5/generation_config.json
[INFO|configuration_utils.py:962] 2026-07-06 17:47:10,518 >> Generate config GenerationConfig {
  "eos_token_id": [
    151644,
    151645,
    151643
  ]
}

07/06/2026 17:47:10 - INFO - __main__ - Finished
07/06/2026 17:47:10 - INFO - __main__ - model.config.force_image_size: 448
07/06/2026 17:47:10 - INFO - __main__ - data_args.force_image_size: 448
07/06/2026 17:47:10 - INFO - __main__ - model.config.vision_config.image_size: 448
07/06/2026 17:47:10 - INFO - __main__ - max_dynamic_patch is set to 12 according to the meta file
07/06/2026 17:47:10 - INFO - __main__ - [Dataset] num_image_token: 256
07/06/2026 17:47:10 - INFO - __main__ - [Dataset] dynamic_image_size: True
07/06/2026 17:47:10 - INFO - __main__ - [Dataset] use_thumbnail: True
07/06/2026 17:47:10 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 12
07/06/2026 17:47:10 - INFO - __main__ - Formatting inputs...Skip in lazy mode
07/06/2026 17:47:11 - INFO - __main__ - Add dataset: vi-envelope-vqa with length: 8843
trainable params: 8,798,208 || all params: 638,496,128 || trainable%: 1.3779579255334184
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.0.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.1.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.2.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.3.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.4.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.5.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.6.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.7.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.8.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.9.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.10.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.11.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.12.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.13.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.14.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.15.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.16.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.17.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.18.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.19.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.20.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.21.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.22.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.k_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.self_attn.o_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.gate_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.up_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_A.default.weight
07/06/2026 17:47:11 - INFO - __main__ - language_model.base_model.model.model.layers.23.mlp.down_proj.lora_B.default.weight
07/06/2026 17:47:11 - INFO - __main__ - mlp1.0.weight
07/06/2026 17:47:11 - INFO - __main__ - mlp1.0.bias
07/06/2026 17:47:11 - INFO - __main__ - mlp1.1.weight
07/06/2026 17:47:11 - INFO - __main__ - mlp1.1.bias
07/06/2026 17:47:11 - INFO - __main__ - mlp1.3.weight
07/06/2026 17:47:11 - INFO - __main__ - mlp1.3.bias
[INFO|trainer.py:641] 2026-07-06 17:47:11,515 >> Using auto half precision backend
[INFO|trainer.py:2078] 2026-07-06 17:47:12,672 >> ***** Running training *****
[INFO|trainer.py:2079] 2026-07-06 17:47:12,673 >>   Num examples = 8,843
[INFO|trainer.py:2080] 2026-07-06 17:47:12,673 >>   Num Epochs = 2
[INFO|trainer.py:2081] 2026-07-06 17:47:12,673 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2084] 2026-07-06 17:47:12,673 >>   Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:2085] 2026-07-06 17:47:12,673 >>   Gradient Accumulation steps = 64
[INFO|trainer.py:2086] 2026-07-06 17:47:12,673 >>   Total optimization steps = 276
[INFO|trainer.py:2087] 2026-07-06 17:47:12,678 >>   Number of trainable parameters = 13,281,024
[INFO|integration_utils.py:723] 2026-07-06 17:47:12,682 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
wandb: Currently logged in as: hhung2662000 (hhung26062000) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run shbvg7yy
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in /root/ai-project/Vintern/internvl_chat/wandb/run-20260706_174713-shbvg7yy
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run vintern_lora_card
wandb: ⭐️ View project at https://wandb.ai/hhung26062000/huggingface
wandb: 🚀 View run at https://wandb.ai/hhung26062000/huggingface/runs/shbvg7yy

  0%|          | 0/276 [00:00<?, ?it/s]/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.10 | optimizer_gradients: 13.07 | optimizer_step: 10.01
[Rank 0] time (ms) | fwd_microstep: 94278.49 | bwd_microstep: 100307.10 | bwd_inner_microstep: 99926.92 | bwd_allreduce_microstep: 100.68 | step_microstep: 69.96
[Rank 0] time (ms) | fwd: 94276.76 | bwd: 100271.07 | bwd_inner: 99919.87 | bwd_allreduce: 99.18 | step: 70.00

  0%|          | 1/276 [03:54<17:56:11, 234.81s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.49 | optimizer_step: 1.02
[Rank 0] time (ms) | fwd_microstep: 92198.33 | bwd_microstep: 99639.72 | bwd_inner_microstep: 99422.43 | bwd_allreduce_microstep: 60.79 | step_microstep: 22.74
[Rank 0] time (ms) | fwd: 92196.58 | bwd: 99613.95 | bwd_inner: 99427.68 | bwd_allreduce: 61.51 | step: 22.74

  1%|          | 2/276 [07:15<16:21:13, 214.87s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93745.31 | bwd_microstep: 99039.19 | bwd_inner_microstep: 98729.99 | bwd_allreduce_microstep: 106.38 | step_microstep: 19.28
[Rank 0] time (ms) | fwd: 93743.59 | bwd: 99029.38 | bwd_inner: 98739.23 | bwd_allreduce: 105.65 | step: 19.30

  1%|          | 3/276 [10:37<15:51:13, 209.06s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 4.46 | optimizer_step: 1.08
[Rank 0] time (ms) | fwd_microstep: 93787.31 | bwd_microstep: 99789.60 | bwd_inner_microstep: 99563.87 | bwd_allreduce_microstep: 54.79 | step_microstep: 39.67
[Rank 0] time (ms) | fwd: 93785.39 | bwd: 99758.00 | bwd_inner: 99551.86 | bwd_allreduce: 58.90 | step: 39.71

  1%|▏         | 4/276 [14:00<15:35:34, 206.38s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 93531.59 | bwd_microstep: 100073.40 | bwd_inner_microstep: 99836.20 | bwd_allreduce_microstep: 58.20 | step_microstep: 19.23
[Rank 0] time (ms) | fwd: 93526.73 | bwd: 100044.92 | bwd_inner: 99823.95 | bwd_allreduce: 62.21 | step: 19.23

  2%|▏         | 5/276 [17:22<15:25:24, 204.89s/it]
                                                   
{'loss': 0.2656, 'grad_norm': 0.017299316823482513, 'learning_rate': 0.00011111111111111112, 'epoch': 0.04}

  2%|▏         | 5/276 [17:22<15:25:24, 204.89s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94015.58 | bwd_microstep: 101397.61 | bwd_inner_microstep: 101173.38 | bwd_allreduce_microstep: 65.04 | step_microstep: 16.82
[Rank 0] time (ms) | fwd: 94014.05 | bwd: 101377.93 | bwd_inner: 101177.48 | bwd_allreduce: 64.86 | step: 16.83

  2%|▏         | 6/276 [20:45<15:19:33, 204.35s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.93 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 93409.91 | bwd_microstep: 101644.79 | bwd_inner_microstep: 101350.00 | bwd_allreduce_microstep: 91.89 | step_microstep: 36.44
[Rank 0] time (ms) | fwd: 93408.21 | bwd: 101624.29 | bwd_inner: 101350.88 | bwd_allreduce: 88.49 | step: 36.46

  3%|▎         | 7/276 [24:07<15:11:54, 203.40s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.50 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 93595.41 | bwd_microstep: 101796.56 | bwd_inner_microstep: 101516.08 | bwd_allreduce_microstep: 78.76 | step_microstep: 20.09
[Rank 0] time (ms) | fwd: 93593.97 | bwd: 101790.57 | bwd_inner: 101514.11 | bwd_allreduce: 96.53 | step: 20.10

  3%|▎         | 8/276 [27:28<15:06:02, 202.84s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 4.25 | optimizer_step: 1.10
[Rank 0] time (ms) | fwd_microstep: 93679.16 | bwd_microstep: 100891.82 | bwd_inner_microstep: 100681.89 | bwd_allreduce_microstep: 44.94 | step_microstep: 74.05
[Rank 0] time (ms) | fwd: 93677.25 | bwd: 100839.82 | bwd_inner: 100664.39 | bwd_allreduce: 42.76 | step: 74.01

  3%|▎         | 9/276 [30:52<15:04:31, 203.26s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93443.26 | bwd_microstep: 101278.68 | bwd_inner_microstep: 101138.29 | bwd_allreduce_microstep: 23.72 | step_microstep: 18.51
[Rank 0] time (ms) | fwd: 93441.42 | bwd: 101241.93 | bwd_inner: 101117.72 | bwd_allreduce: 24.03 | step: 18.52

  4%|▎         | 10/276 [34:16<15:01:37, 203.37s/it]
                                                    
{'loss': 0.0503, 'grad_norm': 0.007974148727953434, 'learning_rate': 0.0001999930778307066, 'epoch': 0.07}

  4%|▎         | 10/276 [34:16<15:01:37, 203.37s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 94647.10 | bwd_microstep: 104585.46 | bwd_inner_microstep: 104441.80 | bwd_allreduce_microstep: 25.61 | step_microstep: 18.08
[Rank 0] time (ms) | fwd: 94641.94 | bwd: 104554.37 | bwd_inner: 104414.57 | bwd_allreduce: 26.06 | step: 18.09

  4%|▍         | 11/276 [37:44<15:03:47, 204.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 92559.45 | bwd_microstep: 98667.28 | bwd_inner_microstep: 98356.37 | bwd_allreduce_microstep: 96.25 | step_microstep: 19.77
[Rank 0] time (ms) | fwd: 92557.69 | bwd: 98635.98 | bwd_inner: 98374.75 | bwd_allreduce: 100.43 | step: 19.79

  4%|▍         | 12/276 [41:04<14:54:28, 203.29s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.13 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93215.54 | bwd_microstep: 99587.01 | bwd_inner_microstep: 99305.31 | bwd_allreduce_microstep: 97.67 | step_microstep: 17.57
[Rank 0] time (ms) | fwd: 93213.71 | bwd: 99585.99 | bwd_inner: 99318.56 | bwd_allreduce: 102.97 | step: 17.58

  5%|▍         | 13/276 [44:25<14:48:15, 202.64s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2022
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 94681.60 | bwd_microstep: 104010.64 | bwd_inner_microstep: 103698.33 | bwd_allreduce_microstep: 116.99 | step_microstep: 17.05
[Rank 0] time (ms) | fwd: 94680.27 | bwd: 104023.72 | bwd_inner: 103740.33 | bwd_allreduce: 123.72 | step: 17.06

  5%|▌         | 14/276 [47:49<14:46:29, 203.01s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.08 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 93260.03 | bwd_microstep: 99200.37 | bwd_inner_microstep: 98979.92 | bwd_allreduce_microstep: 72.12 | step_microstep: 17.88
[Rank 0] time (ms) | fwd: 93259.03 | bwd: 99186.43 | bwd_inner: 98977.17 | bwd_allreduce: 66.01 | step: 17.89

  5%|▌         | 15/276 [51:11<14:41:32, 202.65s/it]
                                                    
{'loss': 0.0278, 'grad_norm': 0.0037575585301965475, 'learning_rate': 0.00019975090251507638, 'epoch': 0.11}

  5%|▌         | 15/276 [51:11<14:41:32, 202.65s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 92234.85 | bwd_microstep: 98967.53 | bwd_inner_microstep: 98688.28 | bwd_allreduce_microstep: 102.40 | step_microstep: 17.59
[Rank 0] time (ms) | fwd: 92233.29 | bwd: 98960.01 | bwd_inner: 98712.79 | bwd_allreduce: 108.17 | step: 17.60

  6%|▌         | 16/276 [54:30<14:34:01, 201.70s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.82 | optimizer_step: 0.97
[Rank 0] time (ms) | fwd_microstep: 93977.94 | bwd_microstep: 102084.70 | bwd_inner_microstep: 101772.90 | bwd_allreduce_microstep: 99.26 | step_microstep: 36.73
[Rank 0] time (ms) | fwd: 93976.24 | bwd: 102049.57 | bwd_inner: 101793.74 | bwd_allreduce: 98.94 | step: 36.74

  6%|▌         | 17/276 [57:53<14:32:45, 202.18s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 93993.84 | bwd_microstep: 100236.73 | bwd_inner_microstep: 99909.56 | bwd_allreduce_microstep: 111.89 | step_microstep: 18.36
[Rank 0] time (ms) | fwd: 93992.36 | bwd: 100229.65 | bwd_inner: 99940.21 | bwd_allreduce: 108.83 | step: 18.37

  7%|▋         | 18/276 [1:01:16<14:30:03, 202.34s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 93251.67 | bwd_microstep: 101632.60 | bwd_inner_microstep: 101326.19 | bwd_allreduce_microstep: 119.07 | step_microstep: 20.06
[Rank 0] time (ms) | fwd: 93250.12 | bwd: 101633.60 | bwd_inner: 101353.54 | bwd_allreduce: 126.96 | step: 20.07

  7%|▋         | 19/276 [1:04:37<14:25:09, 201.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.56 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 94224.91 | bwd_microstep: 98962.35 | bwd_inner_microstep: 98606.47 | bwd_allreduce_microstep: 138.30 | step_microstep: 31.05
[Rank 0] time (ms) | fwd: 94223.15 | bwd: 98971.88 | bwd_inner: 98647.78 | bwd_allreduce: 142.43 | step: 31.06

  7%|▋         | 20/276 [1:08:01<14:23:52, 202.47s/it]
                                                      
{'loss': 0.0176, 'grad_norm': 0.002968734595924616, 'learning_rate': 0.00019916357646713006, 'epoch': 0.14}

  7%|▋         | 20/276 [1:08:01<14:23:52, 202.47s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3550
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 94227.05 | bwd_microstep: 100178.31 | bwd_inner_microstep: 99724.06 | bwd_allreduce_microstep: 233.15 | step_microstep: 20.43
[Rank 0] time (ms) | fwd: 94225.57 | bwd: 100132.85 | bwd_inner: 99713.39 | bwd_allreduce: 235.39 | step: 20.44

  8%|▊         | 21/276 [1:11:24<14:20:46, 202.54s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 92859.39 | bwd_microstep: 99939.42 | bwd_inner_microstep: 99557.12 | bwd_allreduce_microstep: 135.32 | step_microstep: 18.88
[Rank 0] time (ms) | fwd: 92857.92 | bwd: 99939.47 | bwd_inner: 99594.08 | bwd_allreduce: 131.99 | step: 18.89

  8%|▊         | 22/276 [1:14:45<14:15:37, 202.12s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94942.45 | bwd_microstep: 102194.41 | bwd_inner_microstep: 101785.89 | bwd_allreduce_microstep: 235.28 | step_microstep: 20.22
[Rank 0] time (ms) | fwd: 94940.91 | bwd: 102194.86 | bwd_inner: 101806.23 | bwd_allreduce: 229.04 | step: 20.21

  8%|▊         | 23/276 [1:18:10<14:15:51, 202.97s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 91961.86 | bwd_microstep: 100698.26 | bwd_inner_microstep: 100407.86 | bwd_allreduce_microstep: 85.09 | step_microstep: 18.93
[Rank 0] time (ms) | fwd: 91960.36 | bwd: 100674.62 | bwd_inner: 100412.43 | bwd_allreduce: 82.67 | step: 18.97

  9%|▊         | 24/276 [1:21:28<14:07:13, 201.72s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 93645.27 | bwd_microstep: 102285.69 | bwd_inner_microstep: 102094.78 | bwd_allreduce_microstep: 67.26 | step_microstep: 17.05
[Rank 0] time (ms) | fwd: 93643.77 | bwd: 102275.35 | bwd_inner: 102089.61 | bwd_allreduce: 61.33 | step: 17.05

  9%|▉         | 25/276 [1:24:50<14:03:54, 201.73s/it]
                                                      
{'loss': 0.0166, 'grad_norm': 0.004215618595480919, 'learning_rate': 0.00019823313190923795, 'epoch': 0.18}

  9%|▉         | 25/276 [1:24:50<14:03:54, 201.73s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.94 | optimizer_step: 1.06
[Rank 0] time (ms) | fwd_microstep: 93327.71 | bwd_microstep: 99782.86 | bwd_inner_microstep: 99415.81 | bwd_allreduce_microstep: 84.64 | step_microstep: 30.67
[Rank 0] time (ms) | fwd: 93326.07 | bwd: 99752.85 | bwd_inner: 99422.53 | bwd_allreduce: 88.92 | step: 30.74

  9%|▉         | 26/276 [1:28:12<14:01:05, 201.86s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.84 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93672.35 | bwd_microstep: 101842.66 | bwd_inner_microstep: 101394.49 | bwd_allreduce_microstep: 67.64 | step_microstep: 35.47
[Rank 0] time (ms) | fwd: 93670.49 | bwd: 101803.89 | bwd_inner: 101375.30 | bwd_allreduce: 245.58 | step: 35.50

 10%|▉         | 27/276 [1:31:37<14:00:44, 202.59s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93362.17 | bwd_microstep: 103587.97 | bwd_inner_microstep: 103200.34 | bwd_allreduce_microstep: 194.24 | step_microstep: 19.91
[Rank 0] time (ms) | fwd: 93360.65 | bwd: 103575.46 | bwd_inner: 103218.13 | bwd_allreduce: 193.39 | step: 19.92

 10%|█         | 28/276 [1:35:00<13:58:21, 202.83s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94133.51 | bwd_microstep: 103271.75 | bwd_inner_microstep: 103067.73 | bwd_allreduce_microstep: 59.13 | step_microstep: 19.25
[Rank 0] time (ms) | fwd: 94131.75 | bwd: 103258.22 | bwd_inner: 103063.49 | bwd_allreduce: 57.44 | step: 19.26

 11%|█         | 29/276 [1:38:26<13:59:09, 203.85s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 92826.24 | bwd_microstep: 101016.81 | bwd_inner_microstep: 100737.12 | bwd_allreduce_microstep: 80.19 | step_microstep: 17.70
[Rank 0] time (ms) | fwd: 92824.55 | bwd: 100997.44 | bwd_inner: 100731.58 | bwd_allreduce: 87.66 | step: 17.71

 11%|█         | 30/276 [1:41:47<13:52:12, 202.98s/it]
                                                      
{'loss': 0.0119, 'grad_norm': 0.0023846502881497145, 'learning_rate': 0.00019696278829718883, 'epoch': 0.22}

 11%|█         | 30/276 [1:41:47<13:52:12, 202.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 92889.62 | bwd_microstep: 100863.12 | bwd_inner_microstep: 100672.36 | bwd_allreduce_microstep: 87.62 | step_microstep: 20.57
[Rank 0] time (ms) | fwd: 92888.14 | bwd: 100843.27 | bwd_inner: 100662.51 | bwd_allreduce: 84.49 | step: 20.58

 11%|█         | 31/276 [1:45:08<13:45:56, 202.27s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.41 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93209.62 | bwd_microstep: 101546.81 | bwd_inner_microstep: 101148.84 | bwd_allreduce_microstep: 227.75 | step_microstep: 27.55
[Rank 0] time (ms) | fwd: 93208.12 | bwd: 101544.60 | bwd_inner: 101171.90 | bwd_allreduce: 226.28 | step: 27.54

 12%|█▏        | 32/276 [1:48:28<13:39:56, 201.62s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93503.94 | bwd_microstep: 99041.67 | bwd_inner_microstep: 98830.61 | bwd_allreduce_microstep: 63.98 | step_microstep: 17.86
[Rank 0] time (ms) | fwd: 93502.24 | bwd: 99026.88 | bwd_inner: 98830.90 | bwd_allreduce: 57.30 | step: 17.89

 12%|█▏        | 33/276 [1:51:49<13:36:22, 201.57s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 92731.04 | bwd_microstep: 98879.45 | bwd_inner_microstep: 98644.70 | bwd_allreduce_microstep: 72.12 | step_microstep: 17.58
[Rank 0] time (ms) | fwd: 92729.25 | bwd: 98862.07 | bwd_inner: 98648.65 | bwd_allreduce: 70.85 | step: 17.59

 12%|█▏        | 34/276 [1:55:10<13:32:01, 201.33s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.12 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 92582.97 | bwd_microstep: 100716.60 | bwd_inner_microstep: 100362.84 | bwd_allreduce_microstep: 87.41 | step_microstep: 16.71
[Rank 0] time (ms) | fwd: 92581.61 | bwd: 100667.39 | bwd_inner: 100352.89 | bwd_allreduce: 88.48 | step: 16.72

 13%|█▎        | 35/276 [1:58:31<13:27:57, 201.15s/it]
                                                      
{'loss': 0.011, 'grad_norm': 0.005997045431286097, 'learning_rate': 0.00019535694118046583, 'epoch': 0.25}

 13%|█▎        | 35/276 [1:58:31<13:27:57, 201.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.27 | optimizer_step: 1.01
[Rank 0] time (ms) | fwd_microstep: 94365.35 | bwd_microstep: 99691.77 | bwd_inner_microstep: 99392.01 | bwd_allreduce_microstep: 106.00 | step_microstep: 59.11
[Rank 0] time (ms) | fwd: 94363.80 | bwd: 99689.27 | bwd_inner: 99411.14 | bwd_allreduce: 105.56 | step: 59.06

 13%|█▎        | 36/276 [2:01:55<13:28:25, 202.11s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 94860.13 | bwd_microstep: 103585.92 | bwd_inner_microstep: 103254.39 | bwd_allreduce_microstep: 136.46 | step_microstep: 19.60
[Rank 0] time (ms) | fwd: 94858.56 | bwd: 103608.98 | bwd_inner: 103277.40 | bwd_allreduce: 143.19 | step: 19.61

 13%|█▎        | 37/276 [2:05:21<13:28:58, 203.09s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.24 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 94391.82 | bwd_microstep: 100360.43 | bwd_inner_microstep: 99987.61 | bwd_allreduce_microstep: 107.78 | step_microstep: 38.48
[Rank 0] time (ms) | fwd: 94389.99 | bwd: 100339.34 | bwd_inner: 100005.08 | bwd_allreduce: 110.91 | step: 38.50

 14%|█▍        | 38/276 [2:08:44<13:25:52, 203.16s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.10 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 93287.19 | bwd_microstep: 101740.23 | bwd_inner_microstep: 101415.31 | bwd_allreduce_microstep: 105.22 | step_microstep: 19.33
[Rank 0] time (ms) | fwd: 93285.81 | bwd: 101723.08 | bwd_inner: 101424.01 | bwd_allreduce: 108.24 | step: 19.38

 14%|█▍        | 39/276 [2:12:05<13:19:32, 202.41s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 969
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3534
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 93220.84 | bwd_microstep: 99262.77 | bwd_inner_microstep: 99020.65 | bwd_allreduce_microstep: 84.22 | step_microstep: 18.27
[Rank 0] time (ms) | fwd: 93219.09 | bwd: 99254.05 | bwd_inner: 99025.64 | bwd_allreduce: 85.15 | step: 18.28

 14%|█▍        | 40/276 [2:15:26<13:15:10, 202.16s/it]
                                                      
{'loss': 0.0106, 'grad_norm': 0.001673985505476594, 'learning_rate': 0.0001934211469930896, 'epoch': 0.29}

 14%|█▍        | 40/276 [2:15:26<13:15:10, 202.16s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.48 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93685.15 | bwd_microstep: 99053.54 | bwd_inner_microstep: 98711.72 | bwd_allreduce_microstep: 97.07 | step_microstep: 18.00
[Rank 0] time (ms) | fwd: 93683.26 | bwd: 99027.81 | bwd_inner: 98710.12 | bwd_allreduce: 95.59 | step: 18.01

 15%|█▍        | 41/276 [2:18:49<13:12:11, 202.26s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94190.88 | bwd_microstep: 101686.82 | bwd_inner_microstep: 101373.18 | bwd_allreduce_microstep: 95.43 | step_microstep: 17.73
[Rank 0] time (ms) | fwd: 94189.43 | bwd: 101675.20 | bwd_inner: 101202.82 | bwd_allreduce: 96.72 | step: 17.74

 15%|█▌        | 42/276 [2:22:12<13:09:41, 202.48s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3539
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 94887.02 | bwd_microstep: 101967.55 | bwd_inner_microstep: 101609.94 | bwd_allreduce_microstep: 150.23 | step_microstep: 19.58
[Rank 0] time (ms) | fwd: 94885.33 | bwd: 101976.03 | bwd_inner: 101634.69 | bwd_allreduce: 152.20 | step: 19.58

 16%|█▌        | 43/276 [2:25:37<13:10:08, 203.47s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.10 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 92507.54 | bwd_microstep: 98493.09 | bwd_inner_microstep: 98183.12 | bwd_allreduce_microstep: 43.79 | step_microstep: 20.19
[Rank 0] time (ms) | fwd: 92505.29 | bwd: 98415.20 | bwd_inner: 98145.64 | bwd_allreduce: 45.50 | step: 20.24

 16%|█▌        | 44/276 [2:29:00<13:05:04, 203.04s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 93074.79 | bwd_microstep: 104034.57 | bwd_inner_microstep: 103903.91 | bwd_allreduce_microstep: 12.96 | step_microstep: 19.19
[Rank 0] time (ms) | fwd: 93072.92 | bwd: 103990.10 | bwd_inner: 103873.45 | bwd_allreduce: 13.01 | step: 19.17

 16%|█▋        | 45/276 [2:32:23<13:02:45, 203.31s/it]
                                                      
{'loss': 0.0131, 'grad_norm': 0.002548367250710726, 'learning_rate': 0.0001911621038276542, 'epoch': 0.33}

 16%|█▋        | 45/276 [2:32:23<13:02:45, 203.31s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1983
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 93851.66 | bwd_microstep: 103414.96 | bwd_inner_microstep: 103211.49 | bwd_allreduce_microstep: 39.50 | step_microstep: 19.47
[Rank 0] time (ms) | fwd: 93850.03 | bwd: 103363.71 | bwd_inner: 103184.46 | bwd_allreduce: 33.60 | step: 19.49

 17%|█▋        | 46/276 [2:35:49<13:02:05, 204.02s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.54 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93795.38 | bwd_microstep: 101380.68 | bwd_inner_microstep: 101099.94 | bwd_allreduce_microstep: 86.10 | step_microstep: 18.44
[Rank 0] time (ms) | fwd: 93793.65 | bwd: 101368.52 | bwd_inner: 101106.55 | bwd_allreduce: 92.32 | step: 18.47

 17%|█▋        | 47/276 [2:39:12<12:57:54, 203.82s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 93994.86 | bwd_microstep: 101800.16 | bwd_inner_microstep: 101571.66 | bwd_allreduce_microstep: 62.19 | step_microstep: 17.17
[Rank 0] time (ms) | fwd: 93993.27 | bwd: 101782.95 | bwd_inner: 101572.59 | bwd_allreduce: 66.47 | step: 17.18

 17%|█▋        | 48/276 [2:42:36<12:54:20, 203.77s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 978
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 92689.70 | bwd_microstep: 99063.99 | bwd_inner_microstep: 98924.68 | bwd_allreduce_microstep: 42.30 | step_microstep: 22.92
[Rank 0] time (ms) | fwd: 92688.09 | bwd: 99057.30 | bwd_inner: 98936.68 | bwd_allreduce: 45.12 | step: 22.95

 18%|█▊        | 49/276 [2:45:54<12:44:09, 201.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.58 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93327.16 | bwd_microstep: 100631.59 | bwd_inner_microstep: 100355.91 | bwd_allreduce_microstep: 80.98 | step_microstep: 22.60
[Rank 0] time (ms) | fwd: 93325.70 | bwd: 100601.74 | bwd_inner: 100353.07 | bwd_allreduce: 79.89 | step: 22.60

 18%|█▊        | 50/276 [2:49:16<12:40:46, 201.98s/it]
                                                      
{'loss': 0.0117, 'grad_norm': 0.0018577200826257467, 'learning_rate': 0.00018858762825907998, 'epoch': 0.36}

 18%|█▊        | 50/276 [2:49:16<12:40:46, 201.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.04 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 93422.94 | bwd_microstep: 99165.94 | bwd_inner_microstep: 98829.01 | bwd_allreduce_microstep: 91.94 | step_microstep: 34.44
[Rank 0] time (ms) | fwd: 93421.25 | bwd: 99154.14 | bwd_inner: 98835.20 | bwd_allreduce: 101.43 | step: 34.44

 18%|█▊        | 51/276 [2:52:37<12:36:28, 201.73s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1956
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 92861.67 | bwd_microstep: 99110.08 | bwd_inner_microstep: 98880.43 | bwd_allreduce_microstep: 48.04 | step_microstep: 17.76
[Rank 0] time (ms) | fwd: 92859.91 | bwd: 99069.01 | bwd_inner: 98861.09 | bwd_allreduce: 48.30 | step: 17.77

 19%|█▉        | 52/276 [2:55:58<12:32:27, 201.55s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 975
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 92229.81 | bwd_microstep: 97045.49 | bwd_inner_microstep: 96647.86 | bwd_allreduce_microstep: 137.80 | step_microstep: 17.90
[Rank 0] time (ms) | fwd: 92228.11 | bwd: 97036.20 | bwd_inner: 96670.54 | bwd_allreduce: 131.12 | step: 17.91

 19%|█▉        | 53/276 [2:59:17<12:25:57, 200.71s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.07 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93724.62 | bwd_microstep: 100045.46 | bwd_inner_microstep: 99810.88 | bwd_allreduce_microstep: 69.94 | step_microstep: 18.82
[Rank 0] time (ms) | fwd: 93722.96 | bwd: 100037.62 | bwd_inner: 99813.20 | bwd_allreduce: 76.09 | step: 18.82

 20%|█▉        | 54/276 [3:02:39<12:24:19, 201.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 93453.86 | bwd_microstep: 101325.08 | bwd_inner_microstep: 101002.27 | bwd_allreduce_microstep: 99.09 | step_microstep: 18.40
[Rank 0] time (ms) | fwd: 93452.41 | bwd: 101503.95 | bwd_inner: 100997.80 | bwd_allreduce: 97.70 | step: 18.41

 20%|█▉        | 55/276 [3:06:01<12:21:07, 201.21s/it]
                                                      
{'loss': 0.0095, 'grad_norm': 0.0021968537475913763, 'learning_rate': 0.0001857066282982763, 'epoch': 0.4}

 20%|█▉        | 55/276 [3:06:01<12:21:07, 201.21s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 972
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.48 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 93654.07 | bwd_microstep: 98744.53 | bwd_inner_microstep: 98493.24 | bwd_allreduce_microstep: 68.16 | step_microstep: 19.49
[Rank 0] time (ms) | fwd: 93648.05 | bwd: 98730.00 | bwd_inner: 98496.59 | bwd_allreduce: 73.15 | step: 19.49

 20%|██        | 56/276 [3:09:22<12:18:07, 201.31s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.89
[Rank 0] time (ms) | fwd_microstep: 92856.63 | bwd_microstep: 101245.17 | bwd_inner_microstep: 100874.74 | bwd_allreduce_microstep: 113.74 | step_microstep: 19.85
[Rank 0] time (ms) | fwd: 92854.71 | bwd: 101225.65 | bwd_inner: 100902.90 | bwd_allreduce: 104.13 | step: 19.85

 21%|██        | 57/276 [3:12:44<12:15:01, 201.38s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.13 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 93328.40 | bwd_microstep: 99584.25 | bwd_inner_microstep: 99335.84 | bwd_allreduce_microstep: 64.79 | step_microstep: 17.80
[Rank 0] time (ms) | fwd: 93326.47 | bwd: 99565.10 | bwd_inner: 99335.14 | bwd_allreduce: 70.67 | step: 17.79

 21%|██        | 58/276 [3:16:05<12:11:32, 201.34s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 93508.06 | bwd_microstep: 101542.16 | bwd_inner_microstep: 101268.87 | bwd_allreduce_microstep: 86.53 | step_microstep: 18.10
[Rank 0] time (ms) | fwd: 93506.17 | bwd: 101535.64 | bwd_inner: 101276.44 | bwd_allreduce: 98.18 | step: 18.11

 21%|██▏       | 59/276 [3:19:29<12:10:45, 202.05s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 979
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1911
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 91916.29 | bwd_microstep: 98078.88 | bwd_inner_microstep: 97645.46 | bwd_allreduce_microstep: 162.92 | step_microstep: 23.75
[Rank 0] time (ms) | fwd: 91914.59 | bwd: 98085.22 | bwd_inner: 97686.40 | bwd_allreduce: 157.96 | step: 23.74

 22%|██▏       | 60/276 [3:22:47<12:03:54, 201.09s/it]
                                                      
{'loss': 0.0137, 'grad_norm': 0.0027511760126799345, 'learning_rate': 0.00018252907256929775, 'epoch': 0.43}

 22%|██▏       | 60/276 [3:22:47<12:03:54, 201.09s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 92751.32 | bwd_microstep: 98049.80 | bwd_inner_microstep: 97696.21 | bwd_allreduce_microstep: 107.68 | step_microstep: 18.49
[Rank 0] time (ms) | fwd: 92749.21 | bwd: 98037.28 | bwd_inner: 97711.12 | bwd_allreduce: 105.23 | step: 18.49

 22%|██▏       | 61/276 [3:26:09<12:01:06, 201.24s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.39 | optimizer_step: 1.02
[Rank 0] time (ms) | fwd_microstep: 95041.30 | bwd_microstep: 98981.53 | bwd_inner_microstep: 98815.11 | bwd_allreduce_microstep: 10.21 | step_microstep: 58.18
[Rank 0] time (ms) | fwd: 95038.89 | bwd: 98926.14 | bwd_inner: 98766.27 | bwd_allreduce: 10.22 | step: 58.24

 22%|██▏       | 62/276 [3:29:36<12:04:08, 203.03s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 93500.04 | bwd_microstep: 97343.33 | bwd_inner_microstep: 97083.52 | bwd_allreduce_microstep: 51.47 | step_microstep: 18.75
[Rank 0] time (ms) | fwd: 93498.00 | bwd: 97298.44 | bwd_inner: 97058.07 | bwd_allreduce: 53.05 | step: 18.76

 23%|██▎       | 63/276 [3:33:00<12:01:58, 203.37s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93331.48 | bwd_microstep: 101477.03 | bwd_inner_microstep: 101323.69 | bwd_allreduce_microstep: 21.47 | step_microstep: 17.71
[Rank 0] time (ms) | fwd: 93329.44 | bwd: 101438.74 | bwd_inner: 101292.08 | bwd_allreduce: 21.61 | step: 17.71

 23%|██▎       | 64/276 [3:36:25<11:59:42, 203.69s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 93222.32 | bwd_microstep: 103058.56 | bwd_inner_microstep: 102755.58 | bwd_allreduce_microstep: 130.34 | step_microstep: 20.46
[Rank 0] time (ms) | fwd: 93220.69 | bwd: 103072.65 | bwd_inner: 102797.09 | bwd_allreduce: 127.35 | step: 20.46

 24%|██▎       | 65/276 [3:39:47<11:55:04, 203.34s/it]
                                                      
{'loss': 0.011, 'grad_norm': 0.0011977142421528697, 'learning_rate': 0.00017906595581664462, 'epoch': 0.47}

 24%|██▎       | 65/276 [3:39:47<11:55:04, 203.34s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94741.39 | bwd_microstep: 102024.98 | bwd_inner_microstep: 101756.97 | bwd_allreduce_microstep: 91.63 | step_microstep: 17.74
[Rank 0] time (ms) | fwd: 94739.84 | bwd: 102014.97 | bwd_inner: 101773.06 | bwd_allreduce: 89.57 | step: 17.75

 24%|██▍       | 66/276 [3:43:12<11:53:24, 203.83s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3536
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.14 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94862.52 | bwd_microstep: 102425.59 | bwd_inner_microstep: 102228.83 | bwd_allreduce_microstep: 53.18 | step_microstep: 17.64
[Rank 0] time (ms) | fwd: 94860.82 | bwd: 102403.98 | bwd_inner: 102225.24 | bwd_allreduce: 49.46 | step: 17.65

 24%|██▍       | 67/276 [3:46:39<11:52:33, 204.56s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93537.29 | bwd_microstep: 102515.75 | bwd_inner_microstep: 102288.75 | bwd_allreduce_microstep: 76.10 | step_microstep: 19.65
[Rank 0] time (ms) | fwd: 93535.55 | bwd: 102503.96 | bwd_inner: 102295.09 | bwd_allreduce: 76.98 | step: 19.67

 25%|██▍       | 68/276 [3:50:02<11:47:43, 204.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 93799.88 | bwd_microstep: 99927.75 | bwd_inner_microstep: 99528.56 | bwd_allreduce_microstep: 133.23 | step_microstep: 17.63
[Rank 0] time (ms) | fwd: 93798.16 | bwd: 99924.66 | bwd_inner: 99557.10 | bwd_allreduce: 135.58 | step: 17.64

 25%|██▌       | 69/276 [3:53:25<11:43:29, 203.91s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.53 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93351.81 | bwd_microstep: 98923.57 | bwd_inner_microstep: 98691.33 | bwd_allreduce_microstep: 54.23 | step_microstep: 20.67
[Rank 0] time (ms) | fwd: 93350.24 | bwd: 98890.94 | bwd_inner: 98677.79 | bwd_allreduce: 53.39 | step: 20.68

 25%|██▌       | 70/276 [3:56:47<11:37:41, 203.21s/it]
                                                      
{'loss': 0.0103, 'grad_norm': 0.0016937833279371262, 'learning_rate': 0.00017532926086205727, 'epoch': 0.51}

 25%|██▌       | 70/276 [3:56:47<11:37:41, 203.21s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.12 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93139.44 | bwd_microstep: 100080.74 | bwd_inner_microstep: 99864.39 | bwd_allreduce_microstep: 54.35 | step_microstep: 19.27
[Rank 0] time (ms) | fwd: 93137.73 | bwd: 100056.94 | bwd_inner: 99852.23 | bwd_allreduce: 54.60 | step: 19.27

 26%|██▌       | 71/276 [4:00:08<11:32:05, 202.56s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 94244.82 | bwd_microstep: 101987.57 | bwd_inner_microstep: 101573.73 | bwd_allreduce_microstep: 144.98 | step_microstep: 18.21
[Rank 0] time (ms) | fwd: 94243.29 | bwd: 101993.32 | bwd_inner: 101613.27 | bwd_allreduce: 149.06 | step: 18.22

 26%|██▌       | 72/276 [4:03:31<11:29:43, 202.86s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 92352.13 | bwd_microstep: 97094.92 | bwd_inner_microstep: 96915.65 | bwd_allreduce_microstep: 35.11 | step_microstep: 17.90
[Rank 0] time (ms) | fwd: 92350.67 | bwd: 97049.99 | bwd_inner: 96889.79 | bwd_allreduce: 33.63 | step: 17.91

 26%|██▋       | 73/276 [4:06:50<11:22:28, 201.71s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3530
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94936.16 | bwd_microstep: 102670.61 | bwd_inner_microstep: 102436.00 | bwd_allreduce_microstep: 105.12 | step_microstep: 18.62
[Rank 0] time (ms) | fwd: 94934.54 | bwd: 102681.52 | bwd_inner: 102466.62 | bwd_allreduce: 99.71 | step: 18.63

 27%|██▋       | 74/276 [4:10:17<11:23:51, 203.13s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.44 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 92793.28 | bwd_microstep: 101182.89 | bwd_inner_microstep: 100916.44 | bwd_allreduce_microstep: 87.03 | step_microstep: 20.37
[Rank 0] time (ms) | fwd: 92791.72 | bwd: 101164.33 | bwd_inner: 100923.97 | bwd_allreduce: 86.72 | step: 20.38

 27%|██▋       | 75/276 [4:13:38<11:18:36, 202.57s/it]
                                                      
{'loss': 0.0091, 'grad_norm': 0.0024764479603618383, 'learning_rate': 0.00017133191714243805, 'epoch': 0.54}

 27%|██▋       | 75/276 [4:13:38<11:18:36, 202.57s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 92914.29 | bwd_microstep: 99290.84 | bwd_inner_microstep: 99022.95 | bwd_allreduce_microstep: 86.04 | step_microstep: 21.06
[Rank 0] time (ms) | fwd: 92912.61 | bwd: 99254.12 | bwd_inner: 99007.36 | bwd_allreduce: 79.84 | step: 21.07

 28%|██▊       | 76/276 [4:16:59<11:13:47, 202.14s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 93304.61 | bwd_microstep: 100931.16 | bwd_inner_microstep: 100628.62 | bwd_allreduce_microstep: 71.95 | step_microstep: 17.38
[Rank 0] time (ms) | fwd: 93303.11 | bwd: 100891.82 | bwd_inner: 100621.11 | bwd_allreduce: 74.78 | step: 17.39

 28%|██▊       | 77/276 [4:20:20<11:09:30, 201.86s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 94302.89 | bwd_microstep: 101056.07 | bwd_inner_microstep: 100786.71 | bwd_allreduce_microstep: 94.51 | step_microstep: 18.42
[Rank 0] time (ms) | fwd: 94301.10 | bwd: 101029.48 | bwd_inner: 100783.54 | bwd_allreduce: 95.57 | step: 18.43

 28%|██▊       | 78/276 [4:23:45<11:08:24, 202.55s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 93361.89 | bwd_microstep: 101013.09 | bwd_inner_microstep: 100851.12 | bwd_allreduce_microstep: 24.39 | step_microstep: 17.67
[Rank 0] time (ms) | fwd: 93359.96 | bwd: 100981.82 | bwd_inner: 100826.52 | bwd_allreduce: 29.29 | step: 17.68

 29%|██▊       | 79/276 [4:27:06<11:04:23, 202.35s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 93159.41 | bwd_microstep: 101733.90 | bwd_inner_microstep: 101566.62 | bwd_allreduce_microstep: 42.57 | step_microstep: 21.36
[Rank 0] time (ms) | fwd: 93157.41 | bwd: 101704.24 | bwd_inner: 101550.03 | bwd_allreduce: 41.34 | step: 21.37

 29%|██▉       | 80/276 [4:30:31<11:03:39, 203.16s/it]
                                                      
{'loss': 0.0104, 'grad_norm': 0.0016686960589140654, 'learning_rate': 0.00016708775597236506, 'epoch': 0.58}

 29%|██▉       | 80/276 [4:30:31<11:03:39, 203.16s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 92795.89 | bwd_microstep: 98562.69 | bwd_inner_microstep: 98297.12 | bwd_allreduce_microstep: 42.32 | step_microstep: 17.35
[Rank 0] time (ms) | fwd: 92793.77 | bwd: 98507.17 | bwd_inner: 98275.84 | bwd_allreduce: 44.81 | step: 17.36

 29%|██▉       | 81/276 [4:33:55<11:00:48, 203.33s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.78 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93520.67 | bwd_microstep: 100446.16 | bwd_inner_microstep: 100157.71 | bwd_allreduce_microstep: 98.34 | step_microstep: 20.15
[Rank 0] time (ms) | fwd: 93519.00 | bwd: 100436.96 | bwd_inner: 100178.23 | bwd_allreduce: 95.83 | step: 20.15

 30%|██▉       | 82/276 [4:37:18<10:56:53, 203.16s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 5, images per sample: 5.0, dynamic token length: 1481
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 92988.84 | bwd_microstep: 100653.91 | bwd_inner_microstep: 100403.95 | bwd_allreduce_microstep: 97.34 | step_microstep: 17.71
[Rank 0] time (ms) | fwd: 92987.31 | bwd: 100660.65 | bwd_inner: 100447.76 | bwd_allreduce: 100.01 | step: 17.72

 30%|███       | 83/276 [4:40:39<10:51:22, 202.50s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.75 | optimizer_step: 1.08
[Rank 0] time (ms) | fwd_microstep: 93504.81 | bwd_microstep: 97861.80 | bwd_inner_microstep: 97515.15 | bwd_allreduce_microstep: 120.80 | step_microstep: 38.58
[Rank 0] time (ms) | fwd: 93502.88 | bwd: 97846.16 | bwd_inner: 97525.21 | bwd_allreduce: 118.50 | step: 38.63

 30%|███       | 84/276 [4:44:01<10:47:42, 202.41s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94451.19 | bwd_microstep: 103147.78 | bwd_inner_microstep: 102988.38 | bwd_allreduce_microstep: 45.77 | step_microstep: 17.63
[Rank 0] time (ms) | fwd: 94449.42 | bwd: 103133.26 | bwd_inner: 102986.24 | bwd_allreduce: 45.74 | step: 17.64

 31%|███       | 85/276 [4:47:26<10:46:42, 203.15s/it]
                                                      
{'loss': 0.0082, 'grad_norm': 0.001507140463218093, 'learning_rate': 0.00016261146268599564, 'epoch': 0.62}

 31%|███       | 85/276 [4:47:26<10:46:42, 203.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3530
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 93779.55 | bwd_microstep: 99109.65 | bwd_inner_microstep: 98681.35 | bwd_allreduce_microstep: 159.30 | step_microstep: 17.42
[Rank 0] time (ms) | fwd: 93777.95 | bwd: 99114.05 | bwd_inner: 98724.56 | bwd_allreduce: 162.21 | step: 17.43

 31%|███       | 86/276 [4:50:49<10:43:00, 203.05s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 92370.96 | bwd_microstep: 100155.54 | bwd_inner_microstep: 99827.06 | bwd_allreduce_microstep: 121.30 | step_microstep: 18.72
[Rank 0] time (ms) | fwd: 92369.19 | bwd: 100153.63 | bwd_inner: 99857.19 | bwd_allreduce: 114.20 | step: 18.74

 32%|███▏      | 87/276 [4:54:08<10:36:15, 201.99s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 92012.87 | bwd_microstep: 100080.18 | bwd_inner_microstep: 99836.05 | bwd_allreduce_microstep: 77.01 | step_microstep: 17.15
[Rank 0] time (ms) | fwd: 92011.07 | bwd: 100070.80 | bwd_inner: 99859.67 | bwd_allreduce: 79.24 | step: 17.16

 32%|███▏      | 88/276 [4:57:27<10:30:07, 201.11s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 92566.30 | bwd_microstep: 101336.74 | bwd_inner_microstep: 101044.71 | bwd_allreduce_microstep: 93.96 | step_microstep: 17.88
[Rank 0] time (ms) | fwd: 92564.64 | bwd: 101305.91 | bwd_inner: 101059.45 | bwd_allreduce: 94.37 | step: 17.88

 32%|███▏      | 89/276 [5:00:49<10:26:48, 201.12s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.94
[Rank 0] time (ms) | fwd_microstep: 93085.03 | bwd_microstep: 101056.33 | bwd_inner_microstep: 100427.45 | bwd_allreduce_microstep: 420.41 | step_microstep: 23.92
[Rank 0] time (ms) | fwd: 93083.42 | bwd: 101029.81 | bwd_inner: 100427.87 | bwd_allreduce: 415.09 | step: 23.91

 33%|███▎      | 90/276 [5:04:10<10:23:34, 201.15s/it]
                                                      
{'loss': 0.009, 'grad_norm': 0.0012602421920746565, 'learning_rate': 0.00015791852582395334, 'epoch': 0.65}

 33%|███▎      | 90/276 [5:04:10<10:23:34, 201.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.49 | optimizer_step: 0.95
[Rank 0] time (ms) | fwd_microstep: 93436.25 | bwd_microstep: 99432.62 | bwd_inner_microstep: 99108.71 | bwd_allreduce_microstep: 62.29 | step_microstep: 40.08
[Rank 0] time (ms) | fwd: 93434.76 | bwd: 99348.11 | bwd_inner: 99074.98 | bwd_allreduce: 64.54 | step: 40.08

 33%|███▎      | 91/276 [5:07:32<10:21:10, 201.46s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93753.47 | bwd_microstep: 101087.65 | bwd_inner_microstep: 100680.80 | bwd_allreduce_microstep: 228.79 | step_microstep: 22.16
[Rank 0] time (ms) | fwd: 93751.76 | bwd: 101079.04 | bwd_inner: 100688.35 | bwd_allreduce: 222.52 | step: 22.16

 33%|███▎      | 92/276 [5:10:53<10:17:31, 201.37s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3527
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.53 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 94655.22 | bwd_microstep: 102335.99 | bwd_inner_microstep: 101943.14 | bwd_allreduce_microstep: 202.28 | step_microstep: 24.92
[Rank 0] time (ms) | fwd: 94653.62 | bwd: 102325.12 | bwd_inner: 101963.33 | bwd_allreduce: 201.85 | step: 24.92

 34%|███▎      | 93/276 [5:14:18<10:17:13, 202.37s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3539
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94300.65 | bwd_microstep: 101261.80 | bwd_inner_microstep: 101124.23 | bwd_allreduce_microstep: 42.48 | step_microstep: 17.74
[Rank 0] time (ms) | fwd: 94298.88 | bwd: 101247.69 | bwd_inner: 101123.35 | bwd_allreduce: 41.96 | step: 17.74

 34%|███▍      | 94/276 [5:17:41<10:14:54, 202.72s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 973
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 92342.33 | bwd_microstep: 100368.63 | bwd_inner_microstep: 100150.10 | bwd_allreduce_microstep: 65.45 | step_microstep: 18.17
[Rank 0] time (ms) | fwd: 92340.54 | bwd: 100345.97 | bwd_inner: 100152.04 | bwd_allreduce: 66.70 | step: 18.16

 34%|███▍      | 95/276 [5:21:00<10:08:16, 201.64s/it]
                                                      
{'loss': 0.01, 'grad_norm': 0.0012762637343257666, 'learning_rate': 0.0001530251835410199, 'epoch': 0.69}

 34%|███▍      | 95/276 [5:21:00<10:08:16, 201.64s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 92901.41 | bwd_microstep: 98942.36 | bwd_inner_microstep: 98671.37 | bwd_allreduce_microstep: 74.53 | step_microstep: 17.76
[Rank 0] time (ms) | fwd: 92899.46 | bwd: 98911.83 | bwd_inner: 98668.71 | bwd_allreduce: 79.63 | step: 17.76

 35%|███▍      | 96/276 [5:24:22<10:05:12, 201.74s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.24 | optimizer_gradients: 7.01 | optimizer_step: 1.89
[Rank 0] time (ms) | fwd_microstep: 93760.89 | bwd_microstep: 101799.87 | bwd_inner_microstep: 101613.57 | bwd_allreduce_microstep: 50.68 | step_microstep: 80.83
[Rank 0] time (ms) | fwd: 93759.12 | bwd: 101776.61 | bwd_inner: 101599.53 | bwd_allreduce: 56.46 | step: 80.93

 35%|███▌      | 97/276 [5:27:47<10:04:43, 202.70s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93529.36 | bwd_microstep: 100963.17 | bwd_inner_microstep: 100695.41 | bwd_allreduce_microstep: 37.39 | step_microstep: 17.87
[Rank 0] time (ms) | fwd: 93527.46 | bwd: 100915.77 | bwd_inner: 100638.64 | bwd_allreduce: 36.43 | step: 17.86

 36%|███▌      | 98/276 [5:31:13<10:03:32, 203.44s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.45 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94888.85 | bwd_microstep: 101842.01 | bwd_inner_microstep: 101638.65 | bwd_allreduce_microstep: 46.78 | step_microstep: 22.41
[Rank 0] time (ms) | fwd: 94886.77 | bwd: 101799.35 | bwd_inner: 101615.89 | bwd_allreduce: 45.43 | step: 22.42

 36%|███▌      | 99/276 [5:34:39<10:03:16, 204.50s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 92402.15 | bwd_microstep: 98639.46 | bwd_inner_microstep: 98272.71 | bwd_allreduce_microstep: 114.29 | step_microstep: 17.65
[Rank 0] time (ms) | fwd: 92400.35 | bwd: 98629.36 | bwd_inner: 98312.56 | bwd_allreduce: 119.63 | step: 17.68

 36%|███▌      | 100/276 [5:38:01<9:56:51, 203.47s/it]
                                                      
{'loss': 0.0067, 'grad_norm': 0.0010322288144379854, 'learning_rate': 0.00014794836742006666, 'epoch': 0.72}

 36%|███▌      | 100/276 [5:38:01<9:56:51, 203.47s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.14 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 94332.50 | bwd_microstep: 100757.59 | bwd_inner_microstep: 100322.82 | bwd_allreduce_microstep: 161.18 | step_microstep: 17.33
[Rank 0] time (ms) | fwd: 94330.70 | bwd: 100768.77 | bwd_inner: 100364.70 | bwd_allreduce: 158.77 | step: 17.34

 37%|███▋      | 101/276 [5:41:24<9:53:49, 203.60s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3540
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.38 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 94353.30 | bwd_microstep: 102706.21 | bwd_inner_microstep: 102503.67 | bwd_allreduce_microstep: 55.19 | step_microstep: 22.17
[Rank 0] time (ms) | fwd: 94351.51 | bwd: 102687.67 | bwd_inner: 102499.28 | bwd_allreduce: 55.01 | step: 22.18

 37%|███▋      | 102/276 [5:44:50<9:52:02, 204.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 92442.64 | bwd_microstep: 100826.81 | bwd_inner_microstep: 100380.82 | bwd_allreduce_microstep: 176.97 | step_microstep: 18.49
[Rank 0] time (ms) | fwd: 92441.12 | bwd: 100853.64 | bwd_inner: 100429.04 | bwd_allreduce: 178.46 | step: 18.50

 37%|███▋      | 103/276 [5:48:10<9:45:27, 203.05s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 94130.53 | bwd_microstep: 99102.29 | bwd_inner_microstep: 98763.98 | bwd_allreduce_microstep: 135.26 | step_microstep: 18.08
[Rank 0] time (ms) | fwd: 94128.75 | bwd: 99103.21 | bwd_inner: 98785.44 | bwd_allreduce: 114.18 | step: 18.11

 38%|███▊      | 104/276 [5:51:32<9:41:15, 202.77s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 92432.37 | bwd_microstep: 99739.97 | bwd_inner_microstep: 99486.24 | bwd_allreduce_microstep: 85.27 | step_microstep: 18.48
[Rank 0] time (ms) | fwd: 92430.34 | bwd: 99722.62 | bwd_inner: 99490.66 | bwd_allreduce: 85.94 | step: 18.51

 38%|███▊      | 105/276 [5:54:53<9:36:05, 202.14s/it]
                                                      
{'loss': 0.0082, 'grad_norm': 0.0019885932561010122, 'learning_rate': 0.0001427056438866376, 'epoch': 0.76}

 38%|███▊      | 105/276 [5:54:53<9:36:05, 202.14s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93395.19 | bwd_microstep: 100695.03 | bwd_inner_microstep: 100504.40 | bwd_allreduce_microstep: 65.00 | step_microstep: 18.71
[Rank 0] time (ms) | fwd: 93393.63 | bwd: 100691.54 | bwd_inner: 100514.56 | bwd_allreduce: 61.87 | step: 18.72

 38%|███▊      | 106/276 [5:58:14<9:31:41, 201.78s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94268.11 | bwd_microstep: 100020.60 | bwd_inner_microstep: 99769.17 | bwd_allreduce_microstep: 78.54 | step_microstep: 17.42
[Rank 0] time (ms) | fwd: 94266.40 | bwd: 100012.51 | bwd_inner: 99786.82 | bwd_allreduce: 78.07 | step: 17.45

 39%|███▉      | 107/276 [6:01:38<9:29:58, 202.36s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 93503.53 | bwd_microstep: 102136.04 | bwd_inner_microstep: 101956.20 | bwd_allreduce_microstep: 50.82 | step_microstep: 19.77
[Rank 0] time (ms) | fwd: 93502.17 | bwd: 102122.10 | bwd_inner: 101945.50 | bwd_allreduce: 48.91 | step: 19.78

 39%|███▉      | 108/276 [6:05:00<9:26:04, 202.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 94211.80 | bwd_microstep: 103270.43 | bwd_inner_microstep: 103043.42 | bwd_allreduce_microstep: 64.36 | step_microstep: 18.38
[Rank 0] time (ms) | fwd: 94210.41 | bwd: 103245.65 | bwd_inner: 103047.24 | bwd_allreduce: 58.63 | step: 18.39

 39%|███▉      | 109/276 [6:08:23<9:24:03, 202.65s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93467.32 | bwd_microstep: 100745.00 | bwd_inner_microstep: 100493.99 | bwd_allreduce_microstep: 88.98 | step_microstep: 17.97
[Rank 0] time (ms) | fwd: 93465.90 | bwd: 100737.46 | bwd_inner: 100506.03 | bwd_allreduce: 88.90 | step: 17.97

 40%|███▉      | 110/276 [6:11:44<9:19:19, 202.17s/it]
                                                      
{'loss': 0.0087, 'grad_norm': 0.001030433108098805, 'learning_rate': 0.00013731515342689654, 'epoch': 0.8}

 40%|███▉      | 110/276 [6:11:44<9:19:19, 202.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.11 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 93279.38 | bwd_microstep: 100187.69 | bwd_inner_microstep: 99885.55 | bwd_allreduce_microstep: 53.23 | step_microstep: 17.49
[Rank 0] time (ms) | fwd: 93277.53 | bwd: 100147.21 | bwd_inner: 99870.70 | bwd_allreduce: 50.57 | step: 17.50

 40%|████      | 111/276 [6:15:06<9:15:30, 202.00s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.20 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 93870.77 | bwd_microstep: 101050.87 | bwd_inner_microstep: 100841.55 | bwd_allreduce_microstep: 60.65 | step_microstep: 22.54
[Rank 0] time (ms) | fwd: 93869.14 | bwd: 101020.13 | bwd_inner: 100830.14 | bwd_allreduce: 51.01 | step: 22.55

 41%|████      | 112/276 [6:18:28<9:12:16, 202.05s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 94203.42 | bwd_microstep: 99312.86 | bwd_inner_microstep: 98949.86 | bwd_allreduce_microstep: 121.99 | step_microstep: 19.19
[Rank 0] time (ms) | fwd: 94201.84 | bwd: 99297.95 | bwd_inner: 98964.61 | bwd_allreduce: 120.60 | step: 19.20

 41%|████      | 113/276 [6:21:52<9:10:25, 202.61s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3528
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.12 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 94444.69 | bwd_microstep: 102617.17 | bwd_inner_microstep: 102348.96 | bwd_allreduce_microstep: 91.46 | step_microstep: 17.08
[Rank 0] time (ms) | fwd: 94440.14 | bwd: 102609.54 | bwd_inner: 102357.28 | bwd_allreduce: 90.63 | step: 17.11

 41%|████▏     | 114/276 [6:25:18<9:09:43, 203.60s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3538
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93902.63 | bwd_microstep: 101667.37 | bwd_inner_microstep: 101414.52 | bwd_allreduce_microstep: 33.74 | step_microstep: 17.42
[Rank 0] time (ms) | fwd: 93900.79 | bwd: 101606.60 | bwd_inner: 101369.81 | bwd_allreduce: 32.44 | step: 17.43

 42%|████▏     | 115/276 [6:28:44<9:08:05, 204.26s/it]
                                                      
{'loss': 0.007, 'grad_norm': 0.0009500011801719666, 'learning_rate': 0.0001317955478192515, 'epoch': 0.83}

 42%|████▏     | 115/276 [6:28:44<9:08:05, 204.26s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3447
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.10 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 94321.41 | bwd_microstep: 101859.59 | bwd_inner_microstep: 101667.81 | bwd_allreduce_microstep: 34.07 | step_microstep: 16.72
[Rank 0] time (ms) | fwd: 94319.50 | bwd: 101824.20 | bwd_inner: 101640.74 | bwd_allreduce: 33.99 | step: 16.73

 42%|████▏     | 116/276 [6:32:09<9:05:52, 204.70s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.70 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 93093.37 | bwd_microstep: 100516.25 | bwd_inner_microstep: 100295.01 | bwd_allreduce_microstep: 24.07 | step_microstep: 21.55
[Rank 0] time (ms) | fwd: 93091.69 | bwd: 100454.59 | bwd_inner: 100239.50 | bwd_allreduce: 24.36 | step: 21.56

 42%|████▏     | 117/276 [6:35:31<9:00:15, 203.87s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 92581.36 | bwd_microstep: 99444.83 | bwd_inner_microstep: 98940.02 | bwd_allreduce_microstep: 265.83 | step_microstep: 18.07
[Rank 0] time (ms) | fwd: 92579.98 | bwd: 99421.76 | bwd_inner: 98942.87 | bwd_allreduce: 254.59 | step: 18.07

 43%|████▎     | 118/276 [6:38:50<8:52:56, 202.38s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 93184.37 | bwd_microstep: 99773.49 | bwd_inner_microstep: 99433.77 | bwd_allreduce_microstep: 80.40 | step_microstep: 16.84
[Rank 0] time (ms) | fwd: 93182.74 | bwd: 99722.25 | bwd_inner: 99425.42 | bwd_allreduce: 80.03 | step: 16.87

 43%|████▎     | 119/276 [6:42:12<8:48:51, 202.11s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 94060.16 | bwd_microstep: 101468.21 | bwd_inner_microstep: 101141.55 | bwd_allreduce_microstep: 134.96 | step_microstep: 18.19
[Rank 0] time (ms) | fwd: 94058.78 | bwd: 101478.18 | bwd_inner: 101184.44 | bwd_allreduce: 141.50 | step: 18.19

 43%|████▎     | 120/276 [6:45:35<8:46:14, 202.40s/it]
                                                      
{'loss': 0.0104, 'grad_norm': 0.0011772976722568274, 'learning_rate': 0.0001261659255968441, 'epoch': 0.87}

 43%|████▎     | 120/276 [6:45:35<8:46:14, 202.40s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3536
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 94223.13 | bwd_microstep: 102499.68 | bwd_inner_microstep: 102199.24 | bwd_allreduce_microstep: 114.99 | step_microstep: 18.21
[Rank 0] time (ms) | fwd: 94221.67 | bwd: 102506.70 | bwd_inner: 102244.30 | bwd_allreduce: 123.98 | step: 18.22

 44%|████▍     | 121/276 [6:48:58<8:43:28, 202.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.96
[Rank 0] time (ms) | fwd_microstep: 93932.20 | bwd_microstep: 99066.53 | bwd_inner_microstep: 98679.28 | bwd_allreduce_microstep: 128.50 | step_microstep: 28.21
[Rank 0] time (ms) | fwd: 93930.31 | bwd: 99061.00 | bwd_inner: 98699.92 | bwd_allreduce: 133.15 | step: 28.27

 44%|████▍     | 122/276 [6:52:20<8:39:14, 202.30s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 95258.27 | bwd_microstep: 102879.69 | bwd_inner_microstep: 102645.52 | bwd_allreduce_microstep: 86.71 | step_microstep: 18.35
[Rank 0] time (ms) | fwd: 95256.44 | bwd: 102873.23 | bwd_inner: 102652.44 | bwd_allreduce: 80.74 | step: 18.38

 45%|████▍     | 123/276 [6:55:45<8:37:54, 203.10s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94800.91 | bwd_microstep: 101585.23 | bwd_inner_microstep: 101364.02 | bwd_allreduce_microstep: 60.37 | step_microstep: 17.80
[Rank 0] time (ms) | fwd: 94799.33 | bwd: 101559.40 | bwd_inner: 101352.27 | bwd_allreduce: 63.70 | step: 17.81

 45%|████▍     | 124/276 [6:59:10<8:36:11, 203.76s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94947.71 | bwd_microstep: 100604.60 | bwd_inner_microstep: 100262.46 | bwd_allreduce_microstep: 92.98 | step_microstep: 18.33
[Rank 0] time (ms) | fwd: 94945.83 | bwd: 100581.71 | bwd_inner: 100269.79 | bwd_allreduce: 100.17 | step: 18.34

 45%|████▌     | 125/276 [7:02:34<8:33:17, 203.96s/it]
                                                      
{'loss': 0.0097, 'grad_norm': 0.0012133896816521883, 'learning_rate': 0.00012044576596421002, 'epoch': 0.9}

 45%|████▌     | 125/276 [7:02:34<8:33:17, 203.96s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
[Rank 0] time (ms) | optimizer_allgather: 0.04 | optimizer_gradients: 5.04 | optimizer_step: 1.45
[Rank 0] time (ms) | fwd_microstep: 94181.42 | bwd_microstep: 99331.12 | bwd_inner_microstep: 98837.44 | bwd_allreduce_microstep: 182.45 | step_microstep: 121.55
[Rank 0] time (ms) | fwd: 94179.90 | bwd: 99311.66 | bwd_inner: 98855.53 | bwd_allreduce: 187.69 | step: 121.50

 46%|████▌     | 126/276 [7:05:56<8:28:16, 203.31s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3526
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94006.71 | bwd_microstep: 102091.69 | bwd_inner_microstep: 101762.31 | bwd_allreduce_microstep: 129.38 | step_microstep: 18.64
[Rank 0] time (ms) | fwd: 94005.26 | bwd: 102112.44 | bwd_inner: 101812.14 | bwd_allreduce: 129.00 | step: 18.65

 46%|████▌     | 127/276 [7:09:19<8:24:18, 203.08s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 92668.66 | bwd_microstep: 100994.32 | bwd_inner_microstep: 100755.32 | bwd_allreduce_microstep: 64.37 | step_microstep: 17.88
[Rank 0] time (ms) | fwd: 92667.25 | bwd: 100966.17 | bwd_inner: 100756.05 | bwd_allreduce: 65.46 | step: 17.89

 46%|████▋     | 128/276 [7:12:39<8:18:41, 202.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93620.07 | bwd_microstep: 99602.04 | bwd_inner_microstep: 99286.30 | bwd_allreduce_microstep: 104.01 | step_microstep: 18.59
[Rank 0] time (ms) | fwd: 93618.44 | bwd: 99588.14 | bwd_inner: 99314.84 | bwd_allreduce: 106.84 | step: 18.60

 47%|████▋     | 129/276 [7:16:01<8:15:48, 202.37s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 92787.15 | bwd_microstep: 100610.21 | bwd_inner_microstep: 100312.70 | bwd_allreduce_microstep: 88.84 | step_microstep: 17.44
[Rank 0] time (ms) | fwd: 92785.40 | bwd: 100593.70 | bwd_inner: 100316.27 | bwd_allreduce: 93.14 | step: 17.45

 47%|████▋     | 130/276 [7:19:22<8:11:26, 201.96s/it]
                                                      
{'loss': 0.0093, 'grad_norm': 0.001016512862406671, 'learning_rate': 0.00011465486139676955, 'epoch': 0.94}

 47%|████▋     | 130/276 [7:19:22<8:11:26, 201.96s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.61 | optimizer_step: 1.02
[Rank 0] time (ms) | fwd_microstep: 93215.31 | bwd_microstep: 100420.81 | bwd_inner_microstep: 100117.41 | bwd_allreduce_microstep: 110.50 | step_microstep: 51.36
[Rank 0] time (ms) | fwd: 93213.49 | bwd: 100407.00 | bwd_inner: 100131.97 | bwd_allreduce: 112.15 | step: 51.36

 47%|████▋     | 131/276 [7:22:45<8:08:08, 201.99s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.04 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 93656.55 | bwd_microstep: 99620.58 | bwd_inner_microstep: 99464.51 | bwd_allreduce_microstep: 24.95 | step_microstep: 45.11
[Rank 0] time (ms) | fwd: 93654.61 | bwd: 99583.10 | bwd_inner: 99448.06 | bwd_allreduce: 25.63 | step: 45.20

 48%|████▊     | 132/276 [7:26:09<8:06:44, 202.81s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 972
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 92770.64 | bwd_microstep: 98182.15 | bwd_inner_microstep: 97983.38 | bwd_allreduce_microstep: 36.00 | step_microstep: 20.19
[Rank 0] time (ms) | fwd: 92761.91 | bwd: 98138.37 | bwd_inner: 97955.46 | bwd_allreduce: 34.11 | step: 20.23

 48%|████▊     | 133/276 [7:29:32<8:03:32, 202.88s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 978
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 91799.43 | bwd_microstep: 100723.05 | bwd_inner_microstep: 100253.06 | bwd_allreduce_microstep: 216.57 | step_microstep: 19.47
[Rank 0] time (ms) | fwd: 91797.87 | bwd: 100686.36 | bwd_inner: 100235.67 | bwd_allreduce: 217.13 | step: 19.47

 49%|████▊     | 134/276 [7:32:53<7:58:29, 202.18s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 93059.53 | bwd_microstep: 100929.08 | bwd_inner_microstep: 100681.81 | bwd_allreduce_microstep: 75.14 | step_microstep: 17.82
[Rank 0] time (ms) | fwd: 93057.79 | bwd: 100917.76 | bwd_inner: 100692.56 | bwd_allreduce: 77.53 | step: 17.83

 49%|████▉     | 135/276 [7:36:14<7:54:40, 201.99s/it]
                                                      
{'loss': 0.0081, 'grad_norm': 0.0013122924137860537, 'learning_rate': 0.00010881324915636019, 'epoch': 0.98}

 49%|████▉     | 135/276 [7:36:14<7:54:40, 201.99s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.82 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 92717.10 | bwd_microstep: 99509.26 | bwd_inner_microstep: 99189.37 | bwd_allreduce_microstep: 92.06 | step_microstep: 30.32
[Rank 0] time (ms) | fwd: 92715.56 | bwd: 99455.61 | bwd_inner: 99162.49 | bwd_allreduce: 87.27 | step: 30.30

 49%|████▉     | 136/276 [7:39:35<7:50:25, 201.61s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 979
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 92508.42 | bwd_microstep: 100836.25 | bwd_inner_microstep: 100581.45 | bwd_allreduce_microstep: 87.52 | step_microstep: 17.46
[Rank 0] time (ms) | fwd: 92506.79 | bwd: 100825.53 | bwd_inner: 100598.45 | bwd_allreduce: 89.12 | step: 17.46

 50%|████▉     | 137/276 [7:42:55<7:46:04, 201.18s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 93818.67 | bwd_microstep: 99336.73 | bwd_inner_microstep: 99192.86 | bwd_allreduce_microstep: 37.58 | step_microstep: 18.33
[Rank 0] time (ms) | fwd: 93817.10 | bwd: 99315.41 | bwd_inner: 99178.95 | bwd_allreduce: 35.03 | step: 18.36

 50%|█████     | 138/276 [7:46:17<7:43:10, 201.38s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/envs/train_vlm/lib/python3.10/site-packages/timm/models/layers/__init__.py:49: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 94277.77 | bwd_microstep: 97996.35 | bwd_inner_microstep: 97770.01 | bwd_allreduce_microstep: 81.94 | step_microstep: 18.39
[Rank 0] time (ms) | fwd: 94275.73 | bwd: 98003.10 | bwd_inner: 97798.77 | bwd_allreduce: 85.22 | step: 18.40

 50%|█████     | 139/276 [7:50:10<8:01:31, 210.88s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.12 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 92872.26 | bwd_microstep: 99309.49 | bwd_inner_microstep: 98963.67 | bwd_allreduce_microstep: 116.82 | step_microstep: 17.09
[Rank 0] time (ms) | fwd: 92870.71 | bwd: 99310.10 | bwd_inner: 98998.97 | bwd_allreduce: 126.20 | step: 17.10

 51%|█████     | 140/276 [7:53:31<7:51:08, 207.85s/it]
                                                      
{'loss': 0.0094, 'grad_norm': 0.0009057120769284666, 'learning_rate': 0.00010294114195977795, 'epoch': 1.01}

 51%|█████     | 140/276 [7:53:31<7:51:08, 207.85s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 93651.43 | bwd_microstep: 99063.98 | bwd_inner_microstep: 98794.86 | bwd_allreduce_microstep: 64.33 | step_microstep: 18.69
[Rank 0] time (ms) | fwd: 93649.49 | bwd: 99035.08 | bwd_inner: 98787.66 | bwd_allreduce: 69.10 | step: 18.69

 51%|█████     | 141/276 [7:56:53<7:43:39, 206.07s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2022
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 94248.75 | bwd_microstep: 101314.58 | bwd_inner_microstep: 101104.23 | bwd_allreduce_microstep: 75.13 | step_microstep: 21.91
[Rank 0] time (ms) | fwd: 94247.28 | bwd: 101310.05 | bwd_inner: 101124.27 | bwd_allreduce: 77.21 | step: 21.93

 51%|█████▏    | 142/276 [8:00:16<7:38:13, 205.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 91993.39 | bwd_microstep: 97768.12 | bwd_inner_microstep: 97565.60 | bwd_allreduce_microstep: 62.96 | step_microstep: 17.86
[Rank 0] time (ms) | fwd: 91991.71 | bwd: 97760.47 | bwd_inner: 97574.05 | bwd_allreduce: 67.08 | step: 17.84

 52%|█████▏    | 143/276 [8:03:34<7:30:10, 203.09s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 94687.38 | bwd_microstep: 99569.83 | bwd_inner_microstep: 99336.29 | bwd_allreduce_microstep: 62.61 | step_microstep: 17.85
[Rank 0] time (ms) | fwd: 94685.85 | bwd: 99542.63 | bwd_inner: 99319.47 | bwd_allreduce: 56.59 | step: 17.89

 52%|█████▏    | 144/276 [8:06:58<7:27:03, 203.21s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3530
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.63 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 94897.53 | bwd_microstep: 100493.67 | bwd_inner_microstep: 100271.90 | bwd_allreduce_microstep: 66.35 | step_microstep: 19.54
[Rank 0] time (ms) | fwd: 94895.80 | bwd: 100495.62 | bwd_inner: 100293.21 | bwd_allreduce: 65.65 | step: 19.54

 53%|█████▎    | 145/276 [8:10:21<7:23:53, 203.31s/it]
                                                      
{'loss': 0.0071, 'grad_norm': 0.0010994576150551438, 'learning_rate': 9.705885804022206e-05, 'epoch': 1.05}

 53%|█████▎    | 145/276 [8:10:21<7:23:53, 203.31s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93066.79 | bwd_microstep: 101363.71 | bwd_inner_microstep: 101071.95 | bwd_allreduce_microstep: 114.75 | step_microstep: 19.64
[Rank 0] time (ms) | fwd: 93065.33 | bwd: 101371.29 | bwd_inner: 101106.80 | bwd_allreduce: 120.85 | step: 19.65

 53%|█████▎    | 146/276 [8:13:42<7:18:43, 202.49s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 92793.53 | bwd_microstep: 99124.31 | bwd_inner_microstep: 98920.92 | bwd_allreduce_microstep: 65.72 | step_microstep: 17.61
[Rank 0] time (ms) | fwd: 92791.95 | bwd: 99110.12 | bwd_inner: 98917.87 | bwd_allreduce: 64.25 | step: 17.62

 53%|█████▎    | 147/276 [8:17:02<7:14:05, 201.90s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 93232.89 | bwd_microstep: 99290.28 | bwd_inner_microstep: 98983.37 | bwd_allreduce_microstep: 115.23 | step_microstep: 17.78
[Rank 0] time (ms) | fwd: 93231.39 | bwd: 99287.03 | bwd_inner: 99010.35 | bwd_allreduce: 111.43 | step: 17.79

 54%|█████▎    | 148/276 [8:20:23<7:09:59, 201.56s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 92712.55 | bwd_microstep: 101615.11 | bwd_inner_microstep: 101473.47 | bwd_allreduce_microstep: 40.69 | step_microstep: 18.23
[Rank 0] time (ms) | fwd: 92710.90 | bwd: 101595.10 | bwd_inner: 101453.34 | bwd_allreduce: 39.92 | step: 18.24

 54%|█████▍    | 149/276 [8:23:46<7:07:31, 201.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93102.27 | bwd_microstep: 104892.68 | bwd_inner_microstep: 104761.30 | bwd_allreduce_microstep: 28.85 | step_microstep: 19.06
[Rank 0] time (ms) | fwd: 93100.68 | bwd: 104873.50 | bwd_inner: 104747.54 | bwd_allreduce: 31.95 | step: 19.08

 54%|█████▍    | 150/276 [8:27:10<7:05:04, 202.42s/it]
                                                      
{'loss': 0.0061, 'grad_norm': 0.0009362389100715518, 'learning_rate': 9.118675084363986e-05, 'epoch': 1.09}

 54%|█████▍    | 150/276 [8:27:10<7:05:04, 202.42s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1984
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.36 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94507.84 | bwd_microstep: 99511.81 | bwd_inner_microstep: 99341.39 | bwd_allreduce_microstep: 39.00 | step_microstep: 19.85
[Rank 0] time (ms) | fwd: 94506.01 | bwd: 99475.21 | bwd_inner: 99305.31 | bwd_allreduce: 38.35 | step: 19.86

 55%|█████▍    | 151/276 [8:30:34<7:03:17, 203.18s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3540
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 94039.42 | bwd_microstep: 101371.09 | bwd_inner_microstep: 101200.30 | bwd_allreduce_microstep: 41.11 | step_microstep: 19.33
[Rank 0] time (ms) | fwd: 94037.78 | bwd: 101344.36 | bwd_inner: 101196.92 | bwd_allreduce: 39.81 | step: 19.34

 55%|█████▌    | 152/276 [8:33:58<6:59:55, 203.19s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.16 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93758.29 | bwd_microstep: 98118.57 | bwd_inner_microstep: 97663.83 | bwd_allreduce_microstep: 160.89 | step_microstep: 16.97
[Rank 0] time (ms) | fwd: 93756.36 | bwd: 98125.82 | bwd_inner: 97719.36 | bwd_allreduce: 160.05 | step: 16.98

 55%|█████▌    | 153/276 [8:37:21<6:56:25, 203.14s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.47 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 92395.50 | bwd_microstep: 101473.22 | bwd_inner_microstep: 101152.71 | bwd_allreduce_microstep: 109.56 | step_microstep: 21.30
[Rank 0] time (ms) | fwd: 92393.82 | bwd: 101466.77 | bwd_inner: 101166.72 | bwd_allreduce: 124.58 | step: 21.29

 56%|█████▌    | 154/276 [8:40:40<6:50:49, 202.04s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 94457.01 | bwd_microstep: 103304.21 | bwd_inner_microstep: 102555.54 | bwd_allreduce_microstep: 547.96 | step_microstep: 28.58
[Rank 0] time (ms) | fwd: 94455.53 | bwd: 103321.84 | bwd_inner: 102614.56 | bwd_allreduce: 545.12 | step: 28.61

 56%|█████▌    | 155/276 [8:44:05<6:48:50, 202.73s/it]
                                                      
{'loss': 0.0067, 'grad_norm': 0.001235711737535894, 'learning_rate': 8.534513860323046e-05, 'epoch': 1.12}

 56%|█████▌    | 155/276 [8:44:05<6:48:50, 202.73s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 93354.92 | bwd_microstep: 101597.04 | bwd_inner_microstep: 101502.42 | bwd_allreduce_microstep: 18.54 | step_microstep: 17.99
[Rank 0] time (ms) | fwd: 93353.33 | bwd: 101572.59 | bwd_inner: 101486.64 | bwd_allreduce: 17.98 | step: 17.99

 57%|█████▋    | 156/276 [8:47:27<6:45:10, 202.59s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.06 | optimizer_gradients: 4.25 | optimizer_step: 1.08
[Rank 0] time (ms) | fwd_microstep: 93851.78 | bwd_microstep: 99132.19 | bwd_inner_microstep: 98822.71 | bwd_allreduce_microstep: 119.78 | step_microstep: 100.54
[Rank 0] time (ms) | fwd: 93850.17 | bwd: 99130.91 | bwd_inner: 98837.23 | bwd_allreduce: 127.67 | step: 100.58

 57%|█████▋    | 157/276 [8:50:50<6:42:00, 202.69s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 979
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 92906.38 | bwd_microstep: 98951.84 | bwd_inner_microstep: 98576.96 | bwd_allreduce_microstep: 120.93 | step_microstep: 21.91
[Rank 0] time (ms) | fwd: 92904.92 | bwd: 98945.85 | bwd_inner: 98632.23 | bwd_allreduce: 118.88 | step: 21.90

 57%|█████▋    | 158/276 [8:54:10<6:37:05, 201.91s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 93713.38 | bwd_microstep: 100483.91 | bwd_inner_microstep: 100344.85 | bwd_allreduce_microstep: 43.28 | step_microstep: 17.87
[Rank 0] time (ms) | fwd: 93711.58 | bwd: 100470.83 | bwd_inner: 100342.32 | bwd_allreduce: 43.13 | step: 17.88

 58%|█████▊    | 159/276 [8:57:33<6:34:17, 202.20s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.55 | optimizer_step: 1.02
[Rank 0] time (ms) | fwd_microstep: 93275.57 | bwd_microstep: 99442.88 | bwd_inner_microstep: 99164.86 | bwd_allreduce_microstep: 81.08 | step_microstep: 47.61
[Rank 0] time (ms) | fwd: 93274.02 | bwd: 99435.12 | bwd_inner: 99159.93 | bwd_allreduce: 79.00 | step: 47.66

 58%|█████▊    | 160/276 [9:00:54<6:30:18, 201.89s/it]
                                                      
{'loss': 0.0065, 'grad_norm': 0.0010548752034083009, 'learning_rate': 7.955423403578998e-05, 'epoch': 1.16}

 58%|█████▊    | 160/276 [9:00:54<6:30:18, 201.89s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3545
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 96342.39 | bwd_microstep: 101724.55 | bwd_inner_microstep: 101236.03 | bwd_allreduce_microstep: 211.73 | step_microstep: 17.99
[Rank 0] time (ms) | fwd: 96340.55 | bwd: 101751.78 | bwd_inner: 101283.46 | bwd_allreduce: 206.98 | step: 18.00

 58%|█████▊    | 161/276 [9:04:22<6:30:21, 203.67s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94560.91 | bwd_microstep: 99675.69 | bwd_inner_microstep: 99459.64 | bwd_allreduce_microstep: 52.79 | step_microstep: 18.38
[Rank 0] time (ms) | fwd: 94558.91 | bwd: 99646.10 | bwd_inner: 99450.58 | bwd_allreduce: 51.46 | step: 18.39

 59%|█████▊    | 162/276 [9:07:46<6:27:08, 203.76s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 92345.65 | bwd_microstep: 98998.34 | bwd_inner_microstep: 98770.41 | bwd_allreduce_microstep: 66.37 | step_microstep: 18.29
[Rank 0] time (ms) | fwd: 92344.29 | bwd: 98983.41 | bwd_inner: 98774.27 | bwd_allreduce: 67.42 | step: 18.30

 59%|█████▉    | 163/276 [9:11:05<6:21:12, 202.41s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 92943.06 | bwd_microstep: 101768.99 | bwd_inner_microstep: 101534.59 | bwd_allreduce_microstep: 91.31 | step_microstep: 18.69
[Rank 0] time (ms) | fwd: 92941.41 | bwd: 101770.76 | bwd_inner: 101556.10 | bwd_allreduce: 92.57 | step: 18.71

 59%|█████▉    | 164/276 [9:14:26<6:17:18, 202.13s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.50 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 93657.99 | bwd_microstep: 98578.66 | bwd_inner_microstep: 97907.14 | bwd_allreduce_microstep: 311.87 | step_microstep: 25.35
[Rank 0] time (ms) | fwd: 93656.13 | bwd: 98572.56 | bwd_inner: 97952.31 | bwd_allreduce: 302.98 | step: 25.34

 60%|█████▉    | 165/276 [9:17:55<6:17:34, 204.10s/it]
                                                      
{'loss': 0.006, 'grad_norm': 0.0010509549174457788, 'learning_rate': 7.383407440315596e-05, 'epoch': 1.19}

 60%|█████▉    | 165/276 [9:17:55<6:17:34, 204.10s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.85 | optimizer_step: 0.95
[Rank 0] time (ms) | fwd_microstep: 97365.20 | bwd_microstep: 98973.30 | bwd_inner_microstep: 97616.94 | bwd_allreduce_microstep: 618.40 | step_microstep: 28.01
[Rank 0] time (ms) | fwd: 97362.57 | bwd: 98926.49 | bwd_inner: 97654.36 | bwd_allreduce: 604.17 | step: 28.02

 60%|██████    | 166/276 [9:21:44<6:27:59, 211.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.99 | optimizer_step: 0.91
[Rank 0] time (ms) | fwd_microstep: 99815.59 | bwd_microstep: 100110.33 | bwd_inner_microstep: 98111.05 | bwd_allreduce_microstep: 1084.46 | step_microstep: 38.45
[Rank 0] time (ms) | fwd: 99810.27 | bwd: 100137.82 | bwd_inner: 98290.43 | bwd_allreduce: 1061.71 | step: 38.43

 61%|██████    | 167/276 [9:25:41<6:38:16, 219.24s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1956
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 99322.45 | bwd_microstep: 98905.04 | bwd_inner_microstep: 97532.39 | bwd_allreduce_microstep: 581.70 | step_microstep: 18.79
[Rank 0] time (ms) | fwd: 99317.51 | bwd: 98890.99 | bwd_inner: 97697.78 | bwd_allreduce: 590.73 | step: 18.79

 61%|██████    | 168/276 [9:29:38<6:44:13, 224.57s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.89 | optimizer_step: 0.94
[Rank 0] time (ms) | fwd_microstep: 95658.46 | bwd_microstep: 93997.07 | bwd_inner_microstep: 92774.64 | bwd_allreduce_microstep: 532.23 | step_microstep: 21.96
[Rank 0] time (ms) | fwd: 95646.51 | bwd: 93910.41 | bwd_inner: 92823.35 | bwd_allreduce: 532.13 | step: 21.96

 61%|██████    | 169/276 [9:33:23<6:40:50, 224.77s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.73 | optimizer_step: 1.04
[Rank 0] time (ms) | fwd_microstep: 97058.12 | bwd_microstep: 94623.09 | bwd_inner_microstep: 93639.14 | bwd_allreduce_microstep: 315.23 | step_microstep: 46.59
[Rank 0] time (ms) | fwd: 97044.00 | bwd: 94630.66 | bwd_inner: 93724.98 | bwd_allreduce: 354.20 | step: 46.63

 62%|██████▏   | 170/276 [9:37:09<6:37:25, 224.95s/it]
                                                      
{'loss': 0.005, 'grad_norm': 0.0009265425032936037, 'learning_rate': 6.820445218074849e-05, 'epoch': 1.23}

 62%|██████▏   | 170/276 [9:37:09<6:37:25, 224.95s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.73 | optimizer_step: 1.11
[Rank 0] time (ms) | fwd_microstep: 96033.18 | bwd_microstep: 96417.78 | bwd_inner_microstep: 95227.01 | bwd_allreduce_microstep: 469.98 | step_microstep: 524.77
[Rank 0] time (ms) | fwd: 96024.48 | bwd: 96405.52 | bwd_inner: 95296.76 | bwd_allreduce: 484.99 | step: 524.77

 62%|██████▏   | 171/276 [9:40:59<6:36:15, 226.43s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 97268.65 | bwd_microstep: 95910.65 | bwd_inner_microstep: 95120.79 | bwd_allreduce_microstep: 242.73 | step_microstep: 18.33
[Rank 0] time (ms) | fwd: 97259.44 | bwd: 95872.00 | bwd_inner: 95118.33 | bwd_allreduce: 247.79 | step: 18.35

 62%|██████▏   | 172/276 [9:44:47<6:33:18, 226.91s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3550
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 95720.48 | bwd_microstep: 99343.22 | bwd_inner_microstep: 98775.53 | bwd_allreduce_microstep: 211.89 | step_microstep: 17.50
[Rank 0] time (ms) | fwd: 95713.63 | bwd: 99301.43 | bwd_inner: 98806.99 | bwd_allreduce: 211.76 | step: 17.51

 63%|██████▎   | 173/276 [9:48:19<6:22:12, 222.65s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3527
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.19 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 93880.69 | bwd_microstep: 100670.34 | bwd_inner_microstep: 100405.24 | bwd_allreduce_microstep: 74.55 | step_microstep: 18.76
[Rank 0] time (ms) | fwd: 93878.68 | bwd: 100652.11 | bwd_inner: 100405.23 | bwd_allreduce: 76.26 | step: 18.77

 63%|██████▎   | 174/276 [9:51:43<6:08:53, 216.99s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 973
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.98 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 94135.98 | bwd_microstep: 97717.25 | bwd_inner_microstep: 97286.95 | bwd_allreduce_microstep: 189.77 | step_microstep: 410.22
[Rank 0] time (ms) | fwd: 94134.09 | bwd: 97698.31 | bwd_inner: 97312.31 | bwd_allreduce: 161.87 | step: 410.17

 63%|██████▎   | 175/276 [9:55:11<6:00:45, 214.31s/it]
                                                      
{'loss': 0.0054, 'grad_norm': 0.0009228007984347641, 'learning_rate': 6.26848465731035e-05, 'epoch': 1.27}

 63%|██████▎   | 175/276 [9:55:11<6:00:45, 214.31s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.78 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 95736.08 | bwd_microstep: 97010.29 | bwd_inner_microstep: 95732.67 | bwd_allreduce_microstep: 389.69 | step_microstep: 387.55
[Rank 0] time (ms) | fwd: 95719.82 | bwd: 96987.12 | bwd_inner: 95846.26 | bwd_allreduce: 418.08 | step: 387.52

 64%|██████▍   | 176/276 [9:59:02<6:05:16, 219.16s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3539
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.52 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 98226.53 | bwd_microstep: 100470.26 | bwd_inner_microstep: 99153.30 | bwd_allreduce_microstep: 611.19 | step_microstep: 28.34
[Rank 0] time (ms) | fwd: 98222.43 | bwd: 100456.44 | bwd_inner: 99289.50 | bwd_allreduce: 570.09 | step: 28.34

 64%|██████▍   | 177/276 [10:02:54<6:07:50, 222.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.55 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 96300.49 | bwd_microstep: 98616.18 | bwd_inner_microstep: 96978.72 | bwd_allreduce_microstep: 695.20 | step_microstep: 21.81
[Rank 0] time (ms) | fwd: 96298.15 | bwd: 98606.44 | bwd_inner: 97095.48 | bwd_allreduce: 707.46 | step: 21.82

 64%|██████▍   | 178/276 [10:06:48<6:09:31, 226.24s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 95102.88 | bwd_microstep: 100232.61 | bwd_inner_microstep: 98561.26 | bwd_allreduce_microstep: 709.89 | step_microstep: 21.71
[Rank 0] time (ms) | fwd: 95101.15 | bwd: 100258.12 | bwd_inner: 98738.35 | bwd_allreduce: 707.92 | step: 21.71

 65%|██████▍   | 179/276 [10:10:40<6:08:59, 228.25s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.77 | optimizer_step: 1.28
[Rank 0] time (ms) | fwd_microstep: 98297.35 | bwd_microstep: 97562.77 | bwd_inner_microstep: 96186.49 | bwd_allreduce_microstep: 499.25 | step_microstep: 325.86
[Rank 0] time (ms) | fwd: 98288.12 | bwd: 97543.63 | bwd_inner: 96353.79 | bwd_allreduce: 473.92 | step: 325.83

 65%|██████▌   | 180/276 [10:14:31<6:06:28, 229.05s/it]
                                                       
{'loss': 0.0077, 'grad_norm': 0.0008026002906262875, 'learning_rate': 5.729435611336239e-05, 'epoch': 1.3}

 65%|██████▌   | 180/276 [10:14:31<6:06:28, 229.05s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.71 | optimizer_step: 1.09
[Rank 0] time (ms) | fwd_microstep: 98452.45 | bwd_microstep: 99277.66 | bwd_inner_microstep: 97772.98 | bwd_allreduce_microstep: 541.20 | step_microstep: 41.85
[Rank 0] time (ms) | fwd: 98455.24 | bwd: 99297.46 | bwd_inner: 97904.78 | bwd_allreduce: 540.48 | step: 41.91

 66%|██████▌   | 181/276 [10:18:28<6:06:07, 231.24s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 96179.32 | bwd_microstep: 97080.59 | bwd_inner_microstep: 95610.09 | bwd_allreduce_microstep: 508.63 | step_microstep: 19.66
[Rank 0] time (ms) | fwd: 96170.43 | bwd: 97047.98 | bwd_inner: 95756.69 | bwd_allreduce: 507.27 | step: 19.66

 66%|██████▌   | 182/276 [10:22:21<6:03:15, 231.87s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 97317.01 | bwd_microstep: 98803.42 | bwd_inner_microstep: 97362.49 | bwd_allreduce_microstep: 510.16 | step_microstep: 21.68
[Rank 0] time (ms) | fwd: 97313.73 | bwd: 98817.50 | bwd_inner: 97503.26 | bwd_allreduce: 527.90 | step: 21.68

 66%|██████▋   | 183/276 [10:26:16<6:00:43, 232.72s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.17 | optimizer_step: 0.94
[Rank 0] time (ms) | fwd_microstep: 96151.07 | bwd_microstep: 99615.85 | bwd_inner_microstep: 98059.79 | bwd_allreduce_microstep: 581.78 | step_microstep: 400.84
[Rank 0] time (ms) | fwd: 96145.83 | bwd: 99805.97 | bwd_inner: 98209.27 | bwd_allreduce: 595.28 | step: 400.85

 67%|██████▋   | 184/276 [10:30:12<5:58:19, 233.69s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3534
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.75 | optimizer_step: 1.01
[Rank 0] time (ms) | fwd_microstep: 98330.64 | bwd_microstep: 100453.77 | bwd_inner_microstep: 99048.36 | bwd_allreduce_microstep: 528.97 | step_microstep: 382.00
[Rank 0] time (ms) | fwd: 98331.42 | bwd: 100459.64 | bwd_inner: 99184.68 | bwd_allreduce: 522.38 | step: 381.99

 67%|██████▋   | 185/276 [10:34:07<5:54:55, 234.02s/it]
                                                       
{'loss': 0.0051, 'grad_norm': 0.0006433480884879827, 'learning_rate': 5.205163257993341e-05, 'epoch': 1.34}

 67%|██████▋   | 185/276 [10:34:07<5:54:55, 234.02s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3537
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.97 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 98078.15 | bwd_microstep: 97811.15 | bwd_inner_microstep: 96044.90 | bwd_allreduce_microstep: 706.43 | step_microstep: 21.88
[Rank 0] time (ms) | fwd: 98075.56 | bwd: 97806.30 | bwd_inner: 96188.01 | bwd_allreduce: 705.99 | step: 21.91

 67%|██████▋   | 186/276 [10:38:04<5:52:24, 234.94s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.96
[Rank 0] time (ms) | fwd_microstep: 96871.65 | bwd_microstep: 99422.77 | bwd_inner_microstep: 97955.85 | bwd_allreduce_microstep: 608.95 | step_microstep: 19.79
[Rank 0] time (ms) | fwd: 96867.32 | bwd: 99416.69 | bwd_inner: 98088.54 | bwd_allreduce: 598.32 | step: 19.79

 68%|██████▊   | 187/276 [10:41:59<5:48:30, 234.95s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.99 | optimizer_step: 0.98
[Rank 0] time (ms) | fwd_microstep: 97234.05 | bwd_microstep: 101711.94 | bwd_inner_microstep: 99824.91 | bwd_allreduce_microstep: 872.44 | step_microstep: 271.33
[Rank 0] time (ms) | fwd: 97224.07 | bwd: 101770.08 | bwd_inner: 100028.74 | bwd_allreduce: 916.17 | step: 269.71

 68%|██████▊   | 188/276 [10:45:53<5:44:14, 234.72s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 969
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3538
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.46 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 97714.58 | bwd_microstep: 100560.73 | bwd_inner_microstep: 98893.12 | bwd_allreduce_microstep: 685.69 | step_microstep: 20.50
[Rank 0] time (ms) | fwd: 97708.72 | bwd: 100561.72 | bwd_inner: 99074.14 | bwd_allreduce: 671.21 | step: 20.50

 68%|██████▊   | 189/276 [10:49:49<5:41:11, 235.30s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.10 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 97619.69 | bwd_microstep: 99901.60 | bwd_inner_microstep: 98483.53 | bwd_allreduce_microstep: 523.22 | step_microstep: 17.55
[Rank 0] time (ms) | fwd: 97603.30 | bwd: 99931.91 | bwd_inner: 98635.89 | bwd_allreduce: 512.06 | step: 17.56

 69%|██████▉   | 190/276 [10:53:44<5:36:48, 234.98s/it]
                                                       
{'loss': 0.0068, 'grad_norm': 0.0009176199091598392, 'learning_rate': 4.697481645898012e-05, 'epoch': 1.38}

 69%|██████▉   | 190/276 [10:53:44<5:36:48, 234.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.89 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 99138.00 | bwd_microstep: 98336.46 | bwd_inner_microstep: 96798.09 | bwd_allreduce_microstep: 692.36 | step_microstep: 20.74
[Rank 0] time (ms) | fwd: 99132.41 | bwd: 98363.32 | bwd_inner: 96924.89 | bwd_allreduce: 687.16 | step: 20.74

 69%|██████▉   | 191/276 [10:57:37<5:32:14, 234.53s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.45 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 96861.61 | bwd_microstep: 98184.86 | bwd_inner_microstep: 96874.52 | bwd_allreduce_microstep: 425.66 | step_microstep: 19.49
[Rank 0] time (ms) | fwd: 96854.05 | bwd: 98178.27 | bwd_inner: 97033.72 | bwd_allreduce: 414.39 | step: 19.50

 70%|██████▉   | 192/276 [11:01:30<5:27:49, 234.16s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 96296.55 | bwd_microstep: 97798.43 | bwd_inner_microstep: 96176.55 | bwd_allreduce_microstep: 810.27 | step_microstep: 18.27
[Rank 0] time (ms) | fwd: 96290.13 | bwd: 97804.24 | bwd_inner: 96290.10 | bwd_allreduce: 806.15 | step: 18.28

 70%|██████▉   | 193/276 [11:05:24<5:23:30, 233.86s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 7.18 | optimizer_gradients: 29.21 | optimizer_step: 39.64
[Rank 0] time (ms) | fwd_microstep: 95472.33 | bwd_microstep: 96782.97 | bwd_inner_microstep: 95216.84 | bwd_allreduce_microstep: 536.24 | step_microstep: 494.98
[Rank 0] time (ms) | fwd: 95469.52 | bwd: 96739.29 | bwd_inner: 95368.65 | bwd_allreduce: 516.82 | step: 495.06

 70%|███████   | 194/276 [11:09:14<5:18:20, 232.93s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 95995.79 | bwd_microstep: 97912.19 | bwd_inner_microstep: 96470.33 | bwd_allreduce_microstep: 601.15 | step_microstep: 19.01
[Rank 0] time (ms) | fwd: 95993.17 | bwd: 97948.87 | bwd_inner: 96619.69 | bwd_allreduce: 623.85 | step: 19.02

 71%|███████   | 195/276 [11:13:06<5:13:55, 232.54s/it]
                                                       
{'loss': 0.0072, 'grad_norm': 0.0009827411267906427, 'learning_rate': 4.2081474176046646e-05, 'epoch': 1.41}

 71%|███████   | 195/276 [11:13:06<5:13:55, 232.54s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.52 | optimizer_step: 0.90
[Rank 0] time (ms) | fwd_microstep: 97083.73 | bwd_microstep: 95159.01 | bwd_inner_microstep: 94024.46 | bwd_allreduce_microstep: 263.59 | step_microstep: 26.31
[Rank 0] time (ms) | fwd: 97069.82 | bwd: 95096.02 | bwd_inner: 94247.27 | bwd_allreduce: 273.83 | step: 26.32

 71%|███████   | 196/276 [11:16:55<5:08:43, 231.55s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.29 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 99198.85 | bwd_microstep: 101361.14 | bwd_inner_microstep: 99722.35 | bwd_allreduce_microstep: 727.34 | step_microstep: 18.56
[Rank 0] time (ms) | fwd: 99197.90 | bwd: 101370.56 | bwd_inner: 99841.58 | bwd_allreduce: 757.12 | step: 18.57

 71%|███████▏  | 197/276 [11:20:53<5:07:15, 233.35s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.32 | optimizer_gradients: 108.72 | optimizer_step: 2.17
[Rank 0] time (ms) | fwd_microstep: 96200.14 | bwd_microstep: 100257.90 | bwd_inner_microstep: 98698.62 | bwd_allreduce_microstep: 569.12 | step_microstep: 394.63
[Rank 0] time (ms) | fwd: 96190.01 | bwd: 100254.75 | bwd_inner: 98885.80 | bwd_allreduce: 538.61 | step: 394.57

 72%|███████▏  | 198/276 [11:24:46<5:03:29, 233.46s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.03 | optimizer_step: 0.95
[Rank 0] time (ms) | fwd_microstep: 95262.76 | bwd_microstep: 98015.59 | bwd_inner_microstep: 96189.51 | bwd_allreduce_microstep: 901.80 | step_microstep: 44.23
[Rank 0] time (ms) | fwd: 95260.90 | bwd: 98209.86 | bwd_inner: 96312.60 | bwd_allreduce: 898.36 | step: 44.19

 72%|███████▏  | 199/276 [11:28:39<4:59:12, 233.15s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1911
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 96020.58 | bwd_microstep: 95388.87 | bwd_inner_microstep: 93861.90 | bwd_allreduce_microstep: 620.23 | step_microstep: 21.01
[Rank 0] time (ms) | fwd: 96015.09 | bwd: 95356.06 | bwd_inner: 93982.06 | bwd_allreduce: 655.73 | step: 21.02

 72%|███████▏  | 200/276 [11:32:28<4:53:39, 231.84s/it]
                                                       
{'loss': 0.0066, 'grad_norm': 0.001098952954635024, 'learning_rate': 3.738853731400439e-05, 'epoch': 1.45}

 72%|███████▏  | 200/276 [11:32:28<4:53:39, 231.84s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.93 | optimizer_step: 1.02
[Rank 0] time (ms) | fwd_microstep: 95727.85 | bwd_microstep: 96633.43 | bwd_inner_microstep: 95093.74 | bwd_allreduce_microstep: 780.79 | step_microstep: 31.43
[Rank 0] time (ms) | fwd: 95725.53 | bwd: 96644.49 | bwd_inner: 95195.77 | bwd_allreduce: 792.84 | step: 31.44

 73%|███████▎  | 201/276 [11:36:17<4:49:00, 231.20s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 95915.85 | bwd_microstep: 97406.68 | bwd_inner_microstep: 95987.48 | bwd_allreduce_microstep: 516.76 | step_microstep: 19.96
[Rank 0] time (ms) | fwd: 95908.03 | bwd: 97392.89 | bwd_inner: 96126.60 | bwd_allreduce: 592.63 | step: 19.97

 73%|███████▎  | 202/276 [11:40:11<4:46:03, 231.94s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.99 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 97546.86 | bwd_microstep: 99544.08 | bwd_inner_microstep: 97859.81 | bwd_allreduce_microstep: 872.93 | step_microstep: 24.06
[Rank 0] time (ms) | fwd: 97543.35 | bwd: 99508.74 | bwd_inner: 97949.65 | bwd_allreduce: 881.66 | step: 24.08

 74%|███████▎  | 203/276 [11:44:04<4:42:31, 232.21s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 96231.75 | bwd_microstep: 96562.90 | bwd_inner_microstep: 95222.69 | bwd_allreduce_microstep: 567.55 | step_microstep: 19.73
[Rank 0] time (ms) | fwd: 96223.72 | bwd: 96567.21 | bwd_inner: 95327.92 | bwd_allreduce: 570.57 | step: 19.73

 74%|███████▍  | 204/276 [11:47:55<4:38:19, 231.94s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.71 | optimizer_step: 0.95
[Rank 0] time (ms) | fwd_microstep: 96553.73 | bwd_microstep: 99859.24 | bwd_inner_microstep: 98362.45 | bwd_allreduce_microstep: 614.38 | step_microstep: 430.54
[Rank 0] time (ms) | fwd: 96551.63 | bwd: 99892.84 | bwd_inner: 98487.54 | bwd_allreduce: 505.12 | step: 430.57

 74%|███████▍  | 205/276 [11:51:50<4:35:37, 232.92s/it]
                                                       
{'loss': 0.0069, 'grad_norm': 0.001120549044571817, 'learning_rate': 3.291224402763495e-05, 'epoch': 1.48}

 74%|███████▍  | 205/276 [11:51:50<4:35:37, 232.92s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.18 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 97045.33 | bwd_microstep: 98369.33 | bwd_inner_microstep: 97169.49 | bwd_allreduce_microstep: 479.41 | step_microstep: 383.90
[Rank 0] time (ms) | fwd: 97033.73 | bwd: 98357.19 | bwd_inner: 97241.60 | bwd_allreduce: 490.50 | step: 383.83

 75%|███████▍  | 206/276 [11:55:42<4:31:11, 232.45s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 97285.80 | bwd_microstep: 97064.77 | bwd_inner_microstep: 96042.49 | bwd_allreduce_microstep: 387.17 | step_microstep: 18.92
[Rank 0] time (ms) | fwd: 97272.02 | bwd: 97069.65 | bwd_inner: 96103.91 | bwd_allreduce: 391.99 | step: 18.92

 75%|███████▌  | 207/276 [11:59:32<4:26:35, 231.82s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.65 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 98819.69 | bwd_microstep: 98061.47 | bwd_inner_microstep: 96488.19 | bwd_allreduce_microstep: 765.29 | step_microstep: 87.99
[Rank 0] time (ms) | fwd: 98812.39 | bwd: 98072.08 | bwd_inner: 96606.91 | bwd_allreduce: 786.08 | step: 86.06

 75%|███████▌  | 208/276 [12:03:26<4:23:34, 232.56s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3530
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 98735.11 | bwd_microstep: 101487.51 | bwd_inner_microstep: 99878.17 | bwd_allreduce_microstep: 714.38 | step_microstep: 18.84
[Rank 0] time (ms) | fwd: 98726.62 | bwd: 101498.57 | bwd_inner: 100042.58 | bwd_allreduce: 717.29 | step: 18.84

 76%|███████▌  | 209/276 [12:07:22<4:20:48, 233.56s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.48 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 96430.80 | bwd_microstep: 99097.36 | bwd_inner_microstep: 97296.52 | bwd_allreduce_microstep: 886.48 | step_microstep: 24.99
[Rank 0] time (ms) | fwd: 96424.25 | bwd: 99072.38 | bwd_inner: 97394.19 | bwd_allreduce: 893.91 | step: 24.99

 76%|███████▌  | 210/276 [12:11:17<4:17:23, 233.99s/it]
                                                       
{'loss': 0.0058, 'grad_norm': 0.0010113946627825499, 'learning_rate': 2.8668082857562005e-05, 'epoch': 1.52}

 76%|███████▌  | 210/276 [12:11:17<4:17:23, 233.99s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.79 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 98018.07 | bwd_microstep: 101203.04 | bwd_inner_microstep: 99382.43 | bwd_allreduce_microstep: 982.56 | step_microstep: 30.73
[Rank 0] time (ms) | fwd: 98016.36 | bwd: 101193.76 | bwd_inner: 99510.12 | bwd_allreduce: 997.64 | step: 30.70

 76%|███████▋  | 211/276 [12:15:12<4:13:51, 234.33s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 95244.25 | bwd_microstep: 98728.74 | bwd_inner_microstep: 97373.67 | bwd_allreduce_microstep: 568.93 | step_microstep: 21.24
[Rank 0] time (ms) | fwd: 95229.70 | bwd: 98735.45 | bwd_inner: 97514.10 | bwd_allreduce: 602.19 | step: 21.22

 77%|███████▋  | 212/276 [12:19:01<4:08:04, 232.58s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 975
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.06 | optimizer_step: 0.93
[Rank 0] time (ms) | fwd_microstep: 97270.70 | bwd_microstep: 100106.03 | bwd_inner_microstep: 98109.62 | bwd_allreduce_microstep: 1061.02 | step_microstep: 25.19
[Rank 0] time (ms) | fwd: 97255.48 | bwd: 100098.64 | bwd_inner: 98234.88 | bwd_allreduce: 1050.22 | step: 25.20

 77%|███████▋  | 213/276 [12:22:57<4:05:24, 233.73s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 96632.72 | bwd_microstep: 99629.95 | bwd_inner_microstep: 97609.73 | bwd_allreduce_microstep: 945.83 | step_microstep: 19.46
[Rank 0] time (ms) | fwd: 96626.48 | bwd: 99668.76 | bwd_inner: 97801.27 | bwd_allreduce: 956.36 | step: 19.46

 78%|███████▊  | 214/276 [12:26:54<4:02:18, 234.49s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2019
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.48 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 96672.56 | bwd_microstep: 99147.76 | bwd_inner_microstep: 97532.55 | bwd_allreduce_microstep: 755.34 | step_microstep: 21.91
[Rank 0] time (ms) | fwd: 96666.84 | bwd: 99193.83 | bwd_inner: 97650.91 | bwd_allreduce: 738.34 | step: 21.92

 78%|███████▊  | 215/276 [12:30:48<3:58:23, 234.49s/it]
                                                       
{'loss': 0.0065, 'grad_norm': 0.0006556404987350106, 'learning_rate': 2.467073913794272e-05, 'epoch': 1.56}

 78%|███████▊  | 215/276 [12:30:48<3:58:23, 234.49s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 96403.85 | bwd_microstep: 97636.63 | bwd_inner_microstep: 95826.39 | bwd_allreduce_microstep: 862.89 | step_microstep: 21.18
[Rank 0] time (ms) | fwd: 96401.65 | bwd: 97638.41 | bwd_inner: 95954.46 | bwd_allreduce: 875.68 | step: 21.19

 78%|███████▊  | 216/276 [12:34:43<3:54:30, 234.52s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 97209.53 | bwd_microstep: 97854.40 | bwd_inner_microstep: 96608.21 | bwd_allreduce_microstep: 470.52 | step_microstep: 19.39
[Rank 0] time (ms) | fwd: 97199.12 | bwd: 97871.93 | bwd_inner: 96741.46 | bwd_allreduce: 444.06 | step: 19.40

 79%|███████▊  | 217/276 [12:38:34<3:49:36, 233.50s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.30 | optimizer_step: 1.04
[Rank 0] time (ms) | fwd_microstep: 95201.72 | bwd_microstep: 100241.53 | bwd_inner_microstep: 98774.50 | bwd_allreduce_microstep: 526.18 | step_microstep: 390.52
[Rank 0] time (ms) | fwd: 95199.46 | bwd: 100259.44 | bwd_inner: 98982.86 | bwd_allreduce: 544.26 | step: 390.50

 79%|███████▉  | 218/276 [12:42:29<3:46:12, 234.02s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3536
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.34 | optimizer_step: 0.81
[Rank 0] time (ms) | fwd_microstep: 99616.59 | bwd_microstep: 102645.53 | bwd_inner_microstep: 101299.27 | bwd_allreduce_microstep: 504.34 | step_microstep: 18.09
[Rank 0] time (ms) | fwd: 99614.31 | bwd: 102683.30 | bwd_inner: 101470.62 | bwd_allreduce: 485.05 | step: 18.11

 79%|███████▉  | 219/276 [12:46:26<3:43:13, 234.98s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 98485.86 | bwd_microstep: 97145.09 | bwd_inner_microstep: 96150.87 | bwd_allreduce_microstep: 490.20 | step_microstep: 20.20
[Rank 0] time (ms) | fwd: 98475.77 | bwd: 97171.22 | bwd_inner: 96222.66 | bwd_allreduce: 486.27 | step: 20.20

 80%|███████▉  | 220/276 [12:50:18<3:38:29, 234.09s/it]
                                                       
{'loss': 0.0055, 'grad_norm': 0.0013271564384922385, 'learning_rate': 2.0934044183355383e-05, 'epoch': 1.59}

 80%|███████▉  | 220/276 [12:50:18<3:38:29, 234.09s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 96158.98 | bwd_microstep: 94922.48 | bwd_inner_microstep: 93748.61 | bwd_allreduce_microstep: 336.50 | step_microstep: 18.61
[Rank 0] time (ms) | fwd: 96146.17 | bwd: 94889.78 | bwd_inner: 93803.27 | bwd_allreduce: 341.03 | step: 18.62

 80%|████████  | 221/276 [12:54:06<3:32:51, 232.22s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.47 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 96621.00 | bwd_microstep: 98383.93 | bwd_inner_microstep: 96839.41 | bwd_allreduce_microstep: 732.75 | step_microstep: 20.27
[Rank 0] time (ms) | fwd: 96612.02 | bwd: 98367.64 | bwd_inner: 96921.48 | bwd_allreduce: 712.30 | step: 20.29

 80%|████████  | 222/276 [12:57:58<3:28:55, 232.14s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3539
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.49 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 98937.00 | bwd_microstep: 101165.96 | bwd_inner_microstep: 99470.56 | bwd_allreduce_microstep: 682.99 | step_microstep: 21.63
[Rank 0] time (ms) | fwd: 98930.43 | bwd: 101211.45 | bwd_inner: 99634.04 | bwd_allreduce: 691.33 | step: 21.63

 81%|████████  | 223/276 [13:01:56<3:26:37, 233.92s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.61 | optimizer_gradients: 5.98 | optimizer_step: 3.67
[Rank 0] time (ms) | fwd_microstep: 96610.56 | bwd_microstep: 98642.89 | bwd_inner_microstep: 97335.49 | bwd_allreduce_microstep: 465.91 | step_microstep: 231.96
[Rank 0] time (ms) | fwd: 96606.08 | bwd: 98637.55 | bwd_inner: 97405.33 | bwd_allreduce: 451.25 | step: 232.09

 81%|████████  | 224/276 [13:05:51<3:22:57, 234.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.01 | optimizer_gradients: 1.22 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 94862.52 | bwd_microstep: 97666.02 | bwd_inner_microstep: 96235.39 | bwd_allreduce_microstep: 558.65 | step_microstep: 17.95
[Rank 0] time (ms) | fwd: 94860.51 | bwd: 97658.47 | bwd_inner: 96337.46 | bwd_allreduce: 534.56 | step: 17.96

 82%|████████▏ | 225/276 [13:09:43<3:18:36, 233.65s/it]
                                                       
{'loss': 0.0065, 'grad_norm': 0.0009260373772121966, 'learning_rate': 1.7470927430702277e-05, 'epoch': 1.63}

 82%|████████▏ | 225/276 [13:09:43<3:18:36, 233.65s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3536
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 98530.93 | bwd_microstep: 102632.90 | bwd_inner_microstep: 101229.26 | bwd_allreduce_microstep: 648.16 | step_microstep: 19.02
[Rank 0] time (ms) | fwd: 98527.91 | bwd: 102715.03 | bwd_inner: 101389.26 | bwd_allreduce: 634.55 | step: 19.02

 82%|████████▏ | 226/276 [13:13:41<3:15:40, 234.81s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.58 | optimizer_step: 0.96
[Rank 0] time (ms) | fwd_microstep: 95271.68 | bwd_microstep: 98381.16 | bwd_inner_microstep: 96886.76 | bwd_allreduce_microstep: 515.13 | step_microstep: 26.41
[Rank 0] time (ms) | fwd: 95266.69 | bwd: 98378.13 | bwd_inner: 97015.78 | bwd_allreduce: 518.87 | step: 26.45

 82%|████████▏ | 227/276 [13:17:35<3:11:33, 234.57s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 96758.33 | bwd_microstep: 99616.52 | bwd_inner_microstep: 97938.17 | bwd_allreduce_microstep: 683.13 | step_microstep: 20.05
[Rank 0] time (ms) | fwd: 96744.46 | bwd: 99616.25 | bwd_inner: 98055.19 | bwd_allreduce: 697.05 | step: 20.07

 83%|████████▎ | 228/276 [13:21:31<3:08:01, 235.03s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 96775.73 | bwd_microstep: 101810.75 | bwd_inner_microstep: 99950.72 | bwd_allreduce_microstep: 778.10 | step_microstep: 19.86
[Rank 0] time (ms) | fwd: 96773.91 | bwd: 101831.07 | bwd_inner: 100151.02 | bwd_allreduce: 724.18 | step: 19.86

 83%|████████▎ | 229/276 [13:25:30<3:05:02, 236.22s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.51 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 97103.97 | bwd_microstep: 101372.78 | bwd_inner_microstep: 99444.21 | bwd_allreduce_microstep: 1079.43 | step_microstep: 28.10
[Rank 0] time (ms) | fwd: 97098.24 | bwd: 101404.43 | bwd_inner: 99622.54 | bwd_allreduce: 1026.54 | step: 28.06

 83%|████████▎ | 230/276 [13:29:26<3:01:01, 236.11s/it]
                                                       
{'loss': 0.0049, 'grad_norm': 0.0008944201981648803, 'learning_rate': 1.4293371701723702e-05, 'epoch': 1.66}

 83%|████████▎ | 230/276 [13:29:26<3:01:01, 236.11s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.02 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 97512.14 | bwd_microstep: 99673.82 | bwd_inner_microstep: 98383.18 | bwd_allreduce_microstep: 423.98 | step_microstep: 22.59
[Rank 0] time (ms) | fwd: 97507.13 | bwd: 99678.91 | bwd_inner: 98478.03 | bwd_allreduce: 449.05 | step: 22.61

 84%|████████▎ | 231/276 [13:33:21<2:56:46, 235.69s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.35 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 96739.79 | bwd_microstep: 100686.97 | bwd_inner_microstep: 98633.19 | bwd_allreduce_microstep: 1083.61 | step_microstep: 21.12
[Rank 0] time (ms) | fwd: 96722.59 | bwd: 100715.66 | bwd_inner: 98793.34 | bwd_allreduce: 1083.72 | step: 21.13

 84%|████████▍ | 232/276 [13:37:17<2:52:56, 235.83s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.21 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 95626.13 | bwd_microstep: 97763.17 | bwd_inner_microstep: 96492.61 | bwd_allreduce_microstep: 471.12 | step_microstep: 20.31
[Rank 0] time (ms) | fwd: 95621.18 | bwd: 97733.45 | bwd_inner: 96587.61 | bwd_allreduce: 463.06 | step: 20.31

 84%|████████▍ | 233/276 [13:41:06<2:47:33, 233.81s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.26 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94815.92 | bwd_microstep: 100079.86 | bwd_inner_microstep: 98734.83 | bwd_allreduce_microstep: 528.84 | step_microstep: 18.12
[Rank 0] time (ms) | fwd: 94813.93 | bwd: 100119.94 | bwd_inner: 98840.99 | bwd_allreduce: 528.94 | step: 18.12

 85%|████████▍ | 234/276 [13:44:50<2:41:34, 230.82s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2020
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.18 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 93504.88 | bwd_microstep: 98027.37 | bwd_inner_microstep: 97861.27 | bwd_allreduce_microstep: 42.03 | step_microstep: 17.77
[Rank 0] time (ms) | fwd: 93503.13 | bwd: 98000.92 | bwd_inner: 97855.94 | bwd_allreduce: 42.40 | step: 17.78

 85%|████████▌ | 235/276 [13:48:10<2:31:26, 221.61s/it]
                                                       
{'loss': 0.0057, 'grad_norm': 0.0013539205538108945, 'learning_rate': 1.1412371740920036e-05, 'epoch': 1.7}

 85%|████████▌ | 235/276 [13:48:10<2:31:26, 221.61s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.40 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93806.40 | bwd_microstep: 98428.28 | bwd_inner_microstep: 98175.83 | bwd_allreduce_microstep: 100.51 | step_microstep: 18.84
[Rank 0] time (ms) | fwd: 93804.82 | bwd: 98433.28 | bwd_inner: 98207.66 | bwd_allreduce: 98.79 | step: 18.87

 86%|████████▌ | 236/276 [13:51:31<2:23:45, 215.63s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 95602.78 | bwd_microstep: 100538.88 | bwd_inner_microstep: 99954.47 | bwd_allreduce_microstep: 284.68 | step_microstep: 19.83
[Rank 0] time (ms) | fwd: 95600.89 | bwd: 100520.07 | bwd_inner: 99967.72 | bwd_allreduce: 287.89 | step: 19.84

 86%|████████▌ | 237/276 [13:55:00<2:18:52, 213.66s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 972
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 97721.83 | bwd_microstep: 98279.72 | bwd_inner_microstep: 96988.62 | bwd_allreduce_microstep: 488.75 | step_microstep: 20.24
[Rank 0] time (ms) | fwd: 97716.05 | bwd: 98267.49 | bwd_inner: 97122.57 | bwd_allreduce: 494.07 | step: 20.24

 86%|████████▌ | 238/276 [13:58:51<2:18:26, 218.59s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.50 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 96218.56 | bwd_microstep: 99241.53 | bwd_inner_microstep: 97468.82 | bwd_allreduce_microstep: 885.50 | step_microstep: 26.60
[Rank 0] time (ms) | fwd: 96211.30 | bwd: 99216.02 | bwd_inner: 97576.01 | bwd_allreduce: 888.41 | step: 26.60

 87%|████████▋ | 239/276 [14:02:45<2:17:43, 223.33s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.82 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 96731.93 | bwd_microstep: 99193.57 | bwd_inner_microstep: 97324.42 | bwd_allreduce_microstep: 786.74 | step_microstep: 197.75
[Rank 0] time (ms) | fwd: 96725.99 | bwd: 99177.24 | bwd_inner: 97522.85 | bwd_allreduce: 807.37 | step: 196.32

 87%|████████▋ | 240/276 [14:06:38<2:15:46, 226.28s/it]
                                                       
{'loss': 0.007, 'grad_norm': 0.0013567405985668302, 'learning_rate': 8.837896172345827e-06, 'epoch': 1.74}

 87%|████████▋ | 240/276 [14:06:38<2:15:46, 226.28s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 97011.49 | bwd_microstep: 101289.07 | bwd_inner_microstep: 99992.29 | bwd_allreduce_microstep: 446.64 | step_microstep: 17.46
[Rank 0] time (ms) | fwd: 97009.57 | bwd: 101290.56 | bwd_inner: 100139.39 | bwd_allreduce: 433.84 | step: 17.47

 87%|████████▋ | 241/276 [14:10:33<2:13:28, 228.80s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.49 | optimizer_step: 0.82
[Rank 0] time (ms) | fwd_microstep: 98418.23 | bwd_microstep: 103164.26 | bwd_inner_microstep: 101856.51 | bwd_allreduce_microstep: 546.34 | step_microstep: 22.54
[Rank 0] time (ms) | fwd: 98412.43 | bwd: 103168.24 | bwd_inner: 101992.73 | bwd_allreduce: 555.78 | step: 22.55

 88%|████████▊ | 242/276 [14:14:27<2:10:32, 230.37s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 978
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.37 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 99020.26 | bwd_microstep: 98779.72 | bwd_inner_microstep: 97270.88 | bwd_allreduce_microstep: 698.88 | step_microstep: 20.38
[Rank 0] time (ms) | fwd: 99009.20 | bwd: 98783.70 | bwd_inner: 97345.59 | bwd_allreduce: 685.14 | step: 20.40

 88%|████████▊ | 243/276 [14:18:19<2:07:04, 231.06s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 978
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 968
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.22 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 96486.81 | bwd_microstep: 95804.34 | bwd_inner_microstep: 94479.79 | bwd_allreduce_microstep: 473.34 | step_microstep: 17.40
[Rank 0] time (ms) | fwd: 96486.21 | bwd: 95813.72 | bwd_inner: 94595.30 | bwd_allreduce: 462.01 | step: 17.41

 88%|████████▊ | 244/276 [14:22:11<2:03:14, 231.09s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 97347.61 | bwd_microstep: 98948.74 | bwd_inner_microstep: 97664.40 | bwd_allreduce_microstep: 448.89 | step_microstep: 17.52
[Rank 0] time (ms) | fwd: 97342.72 | bwd: 98957.21 | bwd_inner: 97711.74 | bwd_allreduce: 474.89 | step: 17.50

 89%|████████▉ | 245/276 [14:26:07<2:00:08, 232.54s/it]
                                                       
{'loss': 0.0059, 'grad_norm': 0.0009066512575373054, 'learning_rate': 6.578853006910402e-06, 'epoch': 1.77}

 89%|████████▉ | 245/276 [14:26:07<2:00:08, 232.54s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 979
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3526
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.20 | optimizer_step: 0.94
[Rank 0] time (ms) | fwd_microstep: 94253.14 | bwd_microstep: 96668.68 | bwd_inner_microstep: 94769.55 | bwd_allreduce_microstep: 977.26 | step_microstep: 33.52
[Rank 0] time (ms) | fwd: 94246.81 | bwd: 96670.60 | bwd_inner: 94906.45 | bwd_allreduce: 954.53 | step: 33.51

 89%|████████▉ | 246/276 [14:29:55<1:55:42, 231.40s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.13 | optimizer_step: 1.00
[Rank 0] time (ms) | fwd_microstep: 96992.22 | bwd_microstep: 102106.58 | bwd_inner_microstep: 100779.99 | bwd_allreduce_microstep: 522.20 | step_microstep: 409.77
[Rank 0] time (ms) | fwd: 96985.64 | bwd: 102120.78 | bwd_inner: 100913.29 | bwd_allreduce: 478.32 | step: 409.70

 89%|████████▉ | 247/276 [14:33:50<1:52:14, 232.24s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.39 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 98552.81 | bwd_microstep: 102139.61 | bwd_inner_microstep: 100231.60 | bwd_allreduce_microstep: 945.90 | step_microstep: 20.82
[Rank 0] time (ms) | fwd: 98546.45 | bwd: 102139.32 | bwd_inner: 100427.60 | bwd_allreduce: 954.85 | step: 20.82

 90%|████████▉ | 248/276 [14:37:45<1:48:49, 233.19s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 5, images per sample: 5.0, dynamic token length: 1481
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.81 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 99391.05 | bwd_microstep: 102544.06 | bwd_inner_microstep: 101060.45 | bwd_allreduce_microstep: 702.88 | step_microstep: 283.45
[Rank 0] time (ms) | fwd: 99382.59 | bwd: 102581.26 | bwd_inner: 101149.90 | bwd_allreduce: 631.76 | step: 280.92

 90%|█████████ | 249/276 [14:41:36<1:44:42, 232.67s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1986
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.50 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 95727.03 | bwd_microstep: 99459.75 | bwd_inner_microstep: 98252.15 | bwd_allreduce_microstep: 458.74 | step_microstep: 25.95
[Rank 0] time (ms) | fwd: 95717.53 | bwd: 99467.87 | bwd_inner: 98419.98 | bwd_allreduce: 432.23 | step: 25.93

 91%|█████████ | 250/276 [14:45:25<1:40:18, 231.49s/it]
                                                       
{'loss': 0.006, 'grad_norm': 0.0008445039275102317, 'learning_rate': 4.643058819534185e-06, 'epoch': 1.81}

 91%|█████████ | 250/276 [14:45:25<1:40:18, 231.49s/it]dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1985
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 98196.42 | bwd_microstep: 98144.51 | bwd_inner_microstep: 97012.23 | bwd_allreduce_microstep: 482.18 | step_microstep: 19.98
[Rank 0] time (ms) | fwd: 98182.37 | bwd: 98084.77 | bwd_inner: 97033.97 | bwd_allreduce: 516.52 | step: 19.99

 91%|█████████ | 251/276 [14:49:18<1:36:36, 231.87s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 976
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.30 | optimizer_step: 0.78
[Rank 0] time (ms) | fwd_microstep: 94381.47 | bwd_microstep: 97002.49 | bwd_inner_microstep: 95542.36 | bwd_allreduce_microstep: 593.62 | step_microstep: 20.54
[Rank 0] time (ms) | fwd: 94367.69 | bwd: 97021.40 | bwd_inner: 95692.85 | bwd_allreduce: 573.06 | step: 20.54

 91%|█████████▏| 252/276 [14:53:06<1:32:17, 230.72s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.59 | optimizer_step: 1.04
[Rank 0] time (ms) | fwd_microstep: 99241.00 | bwd_microstep: 101314.76 | bwd_inner_microstep: 100028.78 | bwd_allreduce_microstep: 469.61 | step_microstep: 59.48
[Rank 0] time (ms) | fwd: 99238.77 | bwd: 101339.14 | bwd_inner: 100191.40 | bwd_allreduce: 477.93 | step: 59.47

 92%|█████████▏| 253/276 [14:57:04<1:29:19, 233.00s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3528
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.70 | optimizer_step: 0.97
[Rank 0] time (ms) | fwd_microstep: 98351.75 | bwd_microstep: 98241.68 | bwd_inner_microstep: 96683.13 | bwd_allreduce_microstep: 841.22 | step_microstep: 34.30
[Rank 0] time (ms) | fwd: 98345.51 | bwd: 98282.16 | bwd_inner: 96795.94 | bwd_allreduce: 861.72 | step: 34.24

 92%|█████████▏| 254/276 [15:00:58<1:25:28, 233.10s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3532
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.24 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 98577.58 | bwd_microstep: 102003.67 | bwd_inner_microstep: 100573.47 | bwd_allreduce_microstep: 580.94 | step_microstep: 22.04
[Rank 0] time (ms) | fwd: 98573.69 | bwd: 102023.47 | bwd_inner: 100709.20 | bwd_allreduce: 610.45 | step: 22.03

 92%|█████████▏| 255/276 [15:04:57<1:22:12, 234.88s/it]
                                                       
{'loss': 0.0067, 'grad_norm': 0.001150821102783084, 'learning_rate': 3.0372117028111825e-06, 'epoch': 1.85}

 92%|█████████▏| 255/276 [15:04:57<1:22:12, 234.88s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.38 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 96199.83 | bwd_microstep: 100247.54 | bwd_inner_microstep: 98510.27 | bwd_allreduce_microstep: 897.00 | step_microstep: 19.43
[Rank 0] time (ms) | fwd: 96188.24 | bwd: 100258.43 | bwd_inner: 98637.82 | bwd_allreduce: 907.11 | step: 19.43

 93%|█████████▎| 256/276 [15:08:49<1:18:03, 234.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.43 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 98237.76 | bwd_microstep: 96539.97 | bwd_inner_microstep: 95312.04 | bwd_allreduce_microstep: 521.26 | step_microstep: 25.32
[Rank 0] time (ms) | fwd: 98223.67 | bwd: 96526.26 | bwd_inner: 95393.86 | bwd_allreduce: 515.26 | step: 25.31

 93%|█████████▎| 257/276 [15:12:42<1:14:00, 233.71s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.17 | optimizer_step: 0.92
[Rank 0] time (ms) | fwd_microstep: 95390.12 | bwd_microstep: 99930.91 | bwd_inner_microstep: 99023.61 | bwd_allreduce_microstep: 149.00 | step_microstep: 23.78
[Rank 0] time (ms) | fwd: 95388.11 | bwd: 99910.02 | bwd_inner: 99305.86 | bwd_allreduce: 152.34 | step: 23.82

 93%|█████████▎| 258/276 [15:16:14<1:08:11, 227.29s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3447
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.86
[Rank 0] time (ms) | fwd_microstep: 95604.92 | bwd_microstep: 102575.67 | bwd_inner_microstep: 102387.52 | bwd_allreduce_microstep: 53.91 | step_microstep: 18.15
[Rank 0] time (ms) | fwd: 95602.77 | bwd: 102580.95 | bwd_inner: 102396.27 | bwd_allreduce: 54.39 | step: 18.16

 94%|█████████▍| 259/276 [15:19:42<1:02:43, 221.38s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1987
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.06 | optimizer_step: 0.75
[Rank 0] time (ms) | fwd_microstep: 93199.25 | bwd_microstep: 101103.32 | bwd_inner_microstep: 100994.38 | bwd_allreduce_microstep: 10.14 | step_microstep: 17.18
[Rank 0] time (ms) | fwd: 93197.12 | bwd: 101075.15 | bwd_inner: 100930.92 | bwd_allreduce: 10.24 | step: 17.18

 94%|█████████▍| 260/276 [15:23:06<57:40, 216.30s/it]  
                                                     
{'loss': 0.0054, 'grad_norm': 0.0007717085536569357, 'learning_rate': 1.766868090762075e-06, 'epoch': 1.88}

 94%|█████████▍| 260/276 [15:23:06<57:40, 216.30s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.28 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 94744.67 | bwd_microstep: 96372.33 | bwd_inner_microstep: 96094.58 | bwd_allreduce_microstep: 95.54 | step_microstep: 18.26
[Rank 0] time (ms) | fwd: 94740.21 | bwd: 96357.84 | bwd_inner: 96105.86 | bwd_allreduce: 93.10 | step: 18.26

 95%|█████████▍| 261/276 [15:26:30<53:06, 212.45s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 972
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.23 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 93146.45 | bwd_microstep: 98803.61 | bwd_inner_microstep: 98559.03 | bwd_allreduce_microstep: 95.02 | step_microstep: 17.42
[Rank 0] time (ms) | fwd: 93144.97 | bwd: 98803.72 | bwd_inner: 98572.48 | bwd_allreduce: 95.48 | step: 17.42

 95%|█████████▍| 262/276 [15:29:50<48:41, 208.70s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.27 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 92945.04 | bwd_microstep: 99871.34 | bwd_inner_microstep: 99606.31 | bwd_allreduce_microstep: 91.19 | step_microstep: 18.17
[Rank 0] time (ms) | fwd: 92943.26 | bwd: 99860.23 | bwd_inner: 99616.03 | bwd_allreduce: 88.94 | step: 18.17

 95%|█████████▌| 263/276 [15:33:10<44:42, 206.36s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 92988.41 | bwd_microstep: 99989.65 | bwd_inner_microstep: 99698.05 | bwd_allreduce_microstep: 90.32 | step_microstep: 21.36
[Rank 0] time (ms) | fwd: 92986.75 | bwd: 99983.24 | bwd_inner: 99707.22 | bwd_allreduce: 124.44 | step: 21.36

 96%|█████████▌| 264/276 [15:36:31<40:55, 204.66s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.88
[Rank 0] time (ms) | fwd_microstep: 94580.95 | bwd_microstep: 100153.59 | bwd_inner_microstep: 99886.51 | bwd_allreduce_microstep: 90.08 | step_microstep: 18.14
[Rank 0] time (ms) | fwd: 94579.21 | bwd: 100154.62 | bwd_inner: 99897.14 | bwd_allreduce: 97.62 | step: 18.15

 96%|█████████▌| 265/276 [15:39:55<37:28, 204.42s/it]
                                                     
{'loss': 0.0056, 'grad_norm': 0.0008303712238557637, 'learning_rate': 8.364235328699566e-07, 'epoch': 1.92}

 96%|█████████▌| 265/276 [15:39:55<37:28, 204.42s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.85
[Rank 0] time (ms) | fwd_microstep: 93450.58 | bwd_microstep: 100053.20 | bwd_inner_microstep: 99775.27 | bwd_allreduce_microstep: 88.93 | step_microstep: 17.31
[Rank 0] time (ms) | fwd: 93448.98 | bwd: 100043.15 | bwd_inner: 99787.19 | bwd_allreduce: 89.46 | step: 17.31

 96%|█████████▋| 266/276 [15:43:16<33:53, 203.39s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2021
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.31 | optimizer_step: 0.79
[Rank 0] time (ms) | fwd_microstep: 93158.99 | bwd_microstep: 98090.30 | bwd_inner_microstep: 97683.81 | bwd_allreduce_microstep: 250.37 | step_microstep: 20.67
[Rank 0] time (ms) | fwd: 93157.19 | bwd: 98085.54 | bwd_inner: 97665.37 | bwd_allreduce: 251.85 | step: 20.67

 97%|█████████▋| 267/276 [15:46:38<30:25, 202.85s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 2.50 | optimizer_step: 0.87
[Rank 0] time (ms) | fwd_microstep: 93528.00 | bwd_microstep: 101603.60 | bwd_inner_microstep: 101258.98 | bwd_allreduce_microstep: 109.17 | step_microstep: 26.38
[Rank 0] time (ms) | fwd: 93526.52 | bwd: 101599.46 | bwd_inner: 101324.12 | bwd_allreduce: 96.70 | step: 26.44

 97%|█████████▋| 268/276 [15:49:59<27:00, 202.53s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3535
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.42 | optimizer_step: 0.83
[Rank 0] time (ms) | fwd_microstep: 94396.99 | bwd_microstep: 102607.21 | bwd_inner_microstep: 102394.42 | bwd_allreduce_microstep: 78.89 | step_microstep: 19.79
[Rank 0] time (ms) | fwd: 94395.55 | bwd: 102619.55 | bwd_inner: 102423.10 | bwd_allreduce: 83.82 | step: 19.79

 97%|█████████▋| 269/276 [15:53:23<23:39, 202.83s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.33 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94240.39 | bwd_microstep: 100951.16 | bwd_inner_microstep: 100738.69 | bwd_allreduce_microstep: 76.76 | step_microstep: 18.91
[Rank 0] time (ms) | fwd: 94238.91 | bwd: 100949.02 | bwd_inner: 100755.34 | bwd_allreduce: 80.55 | step: 18.92

 98%|█████████▊| 270/276 [15:56:46<20:16, 202.80s/it]
                                                     
{'loss': 0.0048, 'grad_norm': 0.0007604911224916577, 'learning_rate': 2.490974849236216e-07, 'epoch': 1.95}

 98%|█████████▊| 270/276 [15:56:46<20:16, 202.80s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1931
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 3, images per sample: 3.0, dynamic token length: 908
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.20 | optimizer_step: 0.84
[Rank 0] time (ms) | fwd_microstep: 93705.56 | bwd_microstep: 98928.14 | bwd_inner_microstep: 98751.45 | bwd_allreduce_microstep: 34.78 | step_microstep: 17.62
[Rank 0] time (ms) | fwd: 93703.88 | bwd: 98901.98 | bwd_inner: 98728.49 | bwd_allreduce: 38.79 | step: 17.63

 98%|█████████▊| 271/276 [16:00:07<16:52, 202.44s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1983
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3533
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1989
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.17 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 94341.51 | bwd_microstep: 102105.30 | bwd_inner_microstep: 101875.96 | bwd_allreduce_microstep: 80.03 | step_microstep: 17.56
[Rank 0] time (ms) | fwd: 94339.88 | bwd: 102102.06 | bwd_inner: 101886.83 | bwd_allreduce: 94.84 | step: 17.57

 99%|█████████▊| 272/276 [16:03:31<13:31, 202.97s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2010
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 3.41 | optimizer_step: 1.17
[Rank 0] time (ms) | fwd_microstep: 93620.92 | bwd_microstep: 98663.41 | bwd_inner_microstep: 98262.34 | bwd_allreduce_microstep: 156.25 | step_microstep: 41.46
[Rank 0] time (ms) | fwd: 93619.33 | bwd: 98670.43 | bwd_inner: 98317.97 | bwd_allreduce: 160.96 | step: 41.45

 99%|█████████▉| 273/276 [16:06:54<10:08, 202.92s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1988
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 13, images per sample: 13.0, dynamic token length: 3468
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.15 | optimizer_step: 0.77
[Rank 0] time (ms) | fwd_microstep: 95453.18 | bwd_microstep: 103127.61 | bwd_inner_microstep: 102884.20 | bwd_allreduce_microstep: 87.57 | step_microstep: 17.36
[Rank 0] time (ms) | fwd: 95451.63 | bwd: 103122.63 | bwd_inner: 102664.65 | bwd_allreduce: 84.15 | step: 17.36

 99%|█████████▉| 274/276 [16:10:20<06:47, 203.78s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2012
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2015
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2018
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2014
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2004
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2003
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1991
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.25 | optimizer_step: 0.76
[Rank 0] time (ms) | fwd_microstep: 94023.09 | bwd_microstep: 100350.23 | bwd_inner_microstep: 100055.57 | bwd_allreduce_microstep: 84.05 | step_microstep: 17.54
[Rank 0] time (ms) | fwd: 94021.34 | bwd: 100362.36 | bwd_inner: 100097.11 | bwd_allreduce: 113.13 | step: 17.55

100%|█████████▉| 275/276 [16:13:42<03:23, 203.17s/it]
                                                     
{'loss': 0.0065, 'grad_norm': 0.0010599087690934539, 'learning_rate': 6.922169293421821e-09, 'epoch': 1.99}

100%|█████████▉| 275/276 [16:13:42<03:23, 203.17s/it]dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2017
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2001
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2011
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2008
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2013
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2007
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2016
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2002
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2006
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2009
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1997
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2005
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1999
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1996
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1995
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1998
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1992
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 2000
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1994
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1993
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1990
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
dynamic ViT batch size: 7, images per sample: 7.0, dynamic token length: 1932
[Rank 0] time (ms) | optimizer_allgather: 0.00 | optimizer_gradients: 1.32 | optimizer_step: 0.80
[Rank 0] time (ms) | fwd_microstep: 94555.76 | bwd_microstep: 102182.26 | bwd_inner_microstep: 102008.35 | bwd_allreduce_microstep: 31.61 | step_microstep: 17.94
[Rank 0] time (ms) | fwd: 94553.92 | bwd: 102159.14 | bwd_inner: 101995.33 | bwd_allreduce: 31.27 | step: 17.95

100%|██████████| 276/276 [16:17:06<00:00, 203.46s/it]petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
petrel_client is not installed. If you read data locally instead of from ceph, ignore it.
Replace train sampler!!
petrel_client is not installed. Using PIL to load images.
[INFO|trainer.py:2329] 2026-07-07 10:04:22,573 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



                                                     
{'train_runtime': 58629.8958, 'train_samples_per_second': 0.302, 'train_steps_per_second': 0.005, 'train_loss': 0.014006755621833861, 'epoch': 2.0}

100%|██████████| 276/276 [16:17:07<00:00, 203.46s/it]
100%|██████████| 276/276 [16:17:07<00:00, 212.42s/it]
[INFO|trainer.py:3410] 2026-07-07 10:04:24,676 >> Saving model checkpoint to /root/ai-project/Vintern/vlm_training
[INFO|configuration_utils.py:472] 2026-07-07 10:04:24,684 >> Configuration saved in /root/ai-project/Vintern/vlm_training/config.json
[INFO|configuration_utils.py:731] 2026-07-07 10:04:24,686 >> Configuration saved in /root/ai-project/Vintern/vlm_training/generation_config.json
[INFO|modeling_utils.py:2618] 2026-07-07 10:04:32,289 >> Model weights saved in /root/ai-project/Vintern/vlm_training/model.safetensors
[INFO|tokenization_utils_base.py:2513] 2026-07-07 10:04:32,291 >> tokenizer config file saved in /root/ai-project/Vintern/vlm_training/tokenizer_config.json
[INFO|tokenization_utils_base.py:2522] 2026-07-07 10:04:32,292 >> Special tokens file saved in /root/ai-project/Vintern/vlm_training/special_tokens_map.json
[INFO|tokenization_utils_base.py:2573] 2026-07-07 10:04:32,293 >> added tokens file saved in /root/ai-project/Vintern/vlm_training/added_tokens.json
***** train metrics *****
  epoch                    =         1.9975
  total_flos               = 338692201472GF
  train_loss               =          0.014
  train_runtime            =    16:17:09.89
  train_samples            =           8843
  train_samples_per_second =          0.302
  train_steps_per_second   =          0.005
wandb: 
wandb: 🚀 View run vintern_lora_card at: https://wandb.ai/hhung26062000/huggingface/runs/shbvg7yy
wandb: Find logs at: wandb/run-20260706_174713-shbvg7yy/logs
[rank0]:[W707 10:04:36.085715531 ProcessGroupNCCL.cpp:1168] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())