/home/ubuntu/Isaac-GR00T/.venv/lib/python3.10/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
  check_for_updates()
/home/ubuntu/Isaac-GR00T/gr00t/experiment/experiment.py:98: UserWarning: image_crop_size and image_target_size will be deprecated in the future. Please use shortest_image_edge and crop_fraction instead.
  warnings.warn(
05/28/2026 10:18:32 - INFO - Saved config to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg
wandb: Currently logged in as: lucafrat (lucafrat-microsoft) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.23.0
wandb: Run data is saved locally in /home/ubuntu/Isaac-GR00T/wandb/run-20260528_101832-7coniw77
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run groot-wbc-8
wandb: ⭐️ View project at https://wandb.ai/lucafrat-microsoft/groot-wbc
wandb: 🚀 View run at https://wandb.ai/lucafrat-microsoft/groot-wbc/runs/7coniw77
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
/home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:255: FutureWarning: Accessing config attribute `compute_dtype` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'compute_dtype' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.compute_dtype'.
  embedding_dim=self.inner_dim, compute_dtype=self.compute_dtype
/home/ubuntu/Isaac-GR00T/gr00t/model/modules/dit.py:286: FutureWarning: Accessing config attribute `output_dim` directly via 'AlternateVLDiT' object attribute is deprecated. Please access 'output_dim' over 'AlternateVLDiT's config object instead, e.g. 'unet.config.output_dim'.
  self.proj_out_2 = nn.Linear(self.inner_dim, self.output_dim)
Total number of DiT parameters:  1091722240
05/28/2026 10:18:34 - INFO - Using AlternateVLDiT for diffusion model
Total number of SelfAttentionTransformer parameters:  201433088

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:01<00:01,  1.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.79it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.59it/s]
05/28/2026 10:18:38 - INFO - Total parameters: 3,144,016,000
05/28/2026 10:18:38 - INFO - Trainable parameters: 1,620,515,968 (51.54%)

Initializing datasets:   0%|          | 0/1 [00:00<?, ?it/s]Generating stats for /home/ubuntu/groot-files/dataset_wbc_train
Generated 79 shards for dataset /home/ubuntu/groot-files/dataset_wbc_train
Total steps: 80551, average shard length: 1019.632911392405, shard length std: 55.72199920427534

Initializing datasets: 100%|██████████| 1/1 [00:00<00:00, 80.43it/s]
05/28/2026 10:18:43 - INFO - Overriding statistics for embodiment 'unitree_g1_sonic'
05/28/2026 10:18:43 - INFO - Saved dataset statistics for inference
Generated 8 shards for dataset /home/ubuntu/groot-files/dataset_wbc_eval
Total steps: 8147, average shard length: 1018.375, shard length std: 45.367768018715665
05/28/2026 10:18:46 - INFO - Overriding statistics for embodiment 'unitree_g1_sonic'
05/28/2026 10:18:46 - INFO - Held-out eval enabled: /home/ubuntu/groot-files/dataset_wbc_eval every 250 steps
05/28/2026 10:18:46 - INFO - 🚀 Starting training...
05/28/2026 10:18:46 - WARNING - No valid checkpoint found in output directory (/home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8)
Current global step: 0
Creating custom train dataloader

  0%|          | 0/10000 [00:00<?, ?it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 4: Wait for shard 22 in dataset 0 in 17.11 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Wait for shard 50 in dataset 0 in 17.31 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 3: Wait for shard 2 in dataset 0 in 17.73 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 49 in dataset 0 in 19.34 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 0: Wait for shard 53 in dataset 0 in 19.81 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Wait for shard 47 in dataset 0 in 20.05 seconds
Rank 0, Worker 1: Caching shard...
Casting fp32 inputs back to torch.bfloat16 for flash-attn compatibility.
Could not estimate the number of tokens of the input, floating-point operations will not be computed

  0%|          | 1/10000 [00:22<61:57:34, 22.31s/it]
  0%|          | 2/10000 [00:22<25:52:49,  9.32s/it]
  0%|          | 3/10000 [00:22<14:25:16,  5.19s/it]
  0%|          | 4/10000 [00:23<9:02:05,  3.25s/it] 
  0%|          | 5/10000 [00:23<6:03:01,  2.18s/it]
  0%|          | 6/10000 [00:23<4:12:31,  1.52s/it]
  0%|          | 7/10000 [00:23<3:01:40,  1.09s/it]
  0%|          | 8/10000 [00:24<2:14:50,  1.24it/s]
  0%|          | 9/10000 [00:24<1:43:37,  1.61it/s]
  0%|          | 10/10000 [00:24<1:23:42,  1.99it/s]
                                                    
{'loss': 1.1811, 'grad_norm': 0.18529030680656433, 'learning_rate': 1.8e-06}

  0%|          | 10/10000 [00:25<1:23:42,  1.99it/s]
  0%|          | 11/10000 [00:25<1:39:02,  1.68it/s]
  0%|          | 12/10000 [00:25<1:24:20,  1.97it/s]
  0%|          | 13/10000 [00:25<1:11:23,  2.33it/s]
  0%|          | 14/10000 [00:26<1:02:14,  2.67it/s]
  0%|          | 15/10000 [00:26<55:48,  2.98it/s]  
  0%|          | 16/10000 [00:26<49:52,  3.34it/s]
  0%|          | 17/10000 [00:26<45:56,  3.62it/s]
  0%|          | 18/10000 [00:27<44:40,  3.72it/s]
  0%|          | 19/10000 [00:27<44:01,  3.78it/s]
  0%|          | 20/10000 [00:27<44:02,  3.78it/s]
                                                  
{'loss': 1.1821, 'grad_norm': 0.17696067690849304, 'learning_rate': 3.8e-06}

  0%|          | 20/10000 [00:27<44:02,  3.78it/s]
  0%|          | 21/10000 [00:27<43:16,  3.84it/s]
  0%|          | 22/10000 [00:28<42:18,  3.93it/s]
  0%|          | 23/10000 [00:28<41:27,  4.01it/s]
  0%|          | 24/10000 [00:28<40:34,  4.10it/s]
  0%|          | 25/10000 [00:28<39:18,  4.23it/s]
  0%|          | 26/10000 [00:28<39:06,  4.25it/s]
  0%|          | 27/10000 [00:29<39:16,  4.23it/s]
  0%|          | 28/10000 [00:29<41:12,  4.03it/s]
  0%|          | 29/10000 [00:29<42:45,  3.89it/s]
  0%|          | 30/10000 [00:30<43:26,  3.83it/s]
                                                  
{'loss': 1.1694, 'grad_norm': 0.13590584695339203, 'learning_rate': 5.8e-06}

  0%|          | 30/10000 [00:30<43:26,  3.83it/s]
  0%|          | 31/10000 [00:30<41:54,  3.96it/s]
  0%|          | 32/10000 [00:30<39:30,  4.20it/s]
  0%|          | 33/10000 [00:30<37:59,  4.37it/s]
  0%|          | 34/10000 [00:30<36:53,  4.50it/s]
  0%|          | 35/10000 [00:31<36:16,  4.58it/s]
  0%|          | 36/10000 [00:31<37:19,  4.45it/s]
  0%|          | 37/10000 [00:31<37:09,  4.47it/s]
  0%|          | 38/10000 [00:31<38:13,  4.34it/s]
  0%|          | 39/10000 [00:32<38:51,  4.27it/s]
  0%|          | 40/10000 [00:32<37:19,  4.45it/s]
                                                  
{'loss': 1.1571, 'grad_norm': 0.279568612575531, 'learning_rate': 7.8e-06}

  0%|          | 40/10000 [00:32<37:19,  4.45it/s]
  0%|          | 41/10000 [00:32<36:57,  4.49it/s]
  0%|          | 42/10000 [00:32<36:39,  4.53it/s]
  0%|          | 43/10000 [00:32<36:15,  4.58it/s]
  0%|          | 44/10000 [00:33<37:47,  4.39it/s]
  0%|          | 45/10000 [00:33<38:44,  4.28it/s]
  0%|          | 46/10000 [00:33<38:22,  4.32it/s]
  0%|          | 47/10000 [00:33<39:30,  4.20it/s]
  0%|          | 48/10000 [00:34<38:11,  4.34it/s]
  0%|          | 49/10000 [00:34<36:57,  4.49it/s]
  0%|          | 50/10000 [00:34<36:05,  4.59it/s]
                                                  
{'loss': 1.1533, 'grad_norm': 0.43323907256126404, 'learning_rate': 9.800000000000001e-06}

  0%|          | 50/10000 [00:34<36:05,  4.59it/s]
  1%|          | 51/10000 [00:34<36:15,  4.57it/s]
  1%|          | 52/10000 [00:34<36:45,  4.51it/s]
  1%|          | 53/10000 [00:35<38:24,  4.32it/s]
  1%|          | 54/10000 [00:35<39:46,  4.17it/s]
  1%|          | 55/10000 [00:35<40:20,  4.11it/s]
  1%|          | 56/10000 [00:35<39:37,  4.18it/s]
  1%|          | 57/10000 [00:36<37:55,  4.37it/s]
  1%|          | 58/10000 [00:36<37:26,  4.43it/s]
  1%|          | 59/10000 [00:36<36:12,  4.58it/s]
  1%|          | 60/10000 [00:36<35:59,  4.60it/s]
                                                  
{'loss': 1.1449, 'grad_norm': 0.31671810150146484, 'learning_rate': 1.18e-05}

  1%|          | 60/10000 [00:36<35:59,  4.60it/s]
  1%|          | 61/10000 [00:36<35:54,  4.61it/s]
  1%|          | 62/10000 [00:37<35:23,  4.68it/s]
  1%|          | 63/10000 [00:37<35:18,  4.69it/s]
  1%|          | 64/10000 [00:37<34:32,  4.80it/s]
  1%|          | 65/10000 [00:37<34:02,  4.86it/s]
  1%|          | 66/10000 [00:37<33:37,  4.92it/s]
  1%|          | 67/10000 [00:38<33:27,  4.95it/s]
  1%|          | 68/10000 [00:38<32:42,  5.06it/s]
  1%|          | 69/10000 [00:38<32:04,  5.16it/s]
  1%|          | 70/10000 [00:38<31:35,  5.24it/s]
                                                  
{'loss': 1.1184, 'grad_norm': 0.3651115596294403, 'learning_rate': 1.3800000000000002e-05}

  1%|          | 70/10000 [00:38<31:35,  5.24it/s]
  1%|          | 71/10000 [00:38<31:37,  5.23it/s]
  1%|          | 72/10000 [00:39<31:21,  5.28it/s]
  1%|          | 73/10000 [00:39<31:32,  5.25it/s]
  1%|          | 74/10000 [00:39<31:28,  5.26it/s]
  1%|          | 75/10000 [00:39<31:33,  5.24it/s]
  1%|          | 76/10000 [00:39<31:24,  5.27it/s]
  1%|          | 77/10000 [00:40<31:00,  5.33it/s]
  1%|          | 78/10000 [00:40<30:45,  5.38it/s]
  1%|          | 79/10000 [00:40<30:14,  5.47it/s]
  1%|          | 80/10000 [00:40<29:55,  5.53it/s]
                                                  
{'loss': 1.1275, 'grad_norm': 0.44200313091278076, 'learning_rate': 1.58e-05}

  1%|          | 80/10000 [00:40<29:55,  5.53it/s]
  1%|          | 81/10000 [00:40<30:00,  5.51it/s]
  1%|          | 82/10000 [00:40<30:20,  5.45it/s]
  1%|          | 83/10000 [00:41<30:37,  5.40it/s]
  1%|          | 84/10000 [00:41<31:14,  5.29it/s]
  1%|          | 85/10000 [00:41<31:38,  5.22it/s]
  1%|          | 86/10000 [00:41<31:14,  5.29it/s]
  1%|          | 87/10000 [00:41<30:48,  5.36it/s]
  1%|          | 88/10000 [00:42<30:28,  5.42it/s]
  1%|          | 89/10000 [00:42<30:29,  5.42it/s]
  1%|          | 90/10000 [00:42<30:33,  5.41it/s]
                                                  
{'loss': 1.1258, 'grad_norm': 0.5552253723144531, 'learning_rate': 1.78e-05}

  1%|          | 90/10000 [00:42<30:33,  5.41it/s]
  1%|          | 91/10000 [00:42<31:18,  5.28it/s]
  1%|          | 92/10000 [00:42<31:16,  5.28it/s]
  1%|          | 93/10000 [00:43<31:00,  5.32it/s]
  1%|          | 94/10000 [00:43<31:12,  5.29it/s]
  1%|          | 95/10000 [00:43<30:49,  5.36it/s]
  1%|          | 96/10000 [00:43<30:37,  5.39it/s]
  1%|          | 97/10000 [00:43<30:44,  5.37it/s]
  1%|          | 98/10000 [00:43<30:43,  5.37it/s]
  1%|          | 99/10000 [00:44<30:47,  5.36it/s]
  1%|          | 100/10000 [00:44<31:22,  5.26it/s]
                                                   
{'loss': 1.1249, 'grad_norm': 0.3365529477596283, 'learning_rate': 1.9800000000000004e-05}

  1%|          | 100/10000 [00:44<31:22,  5.26it/s]
  1%|          | 101/10000 [00:44<32:02,  5.15it/s]
  1%|          | 102/10000 [00:44<32:06,  5.14it/s]
  1%|          | 103/10000 [00:44<32:16,  5.11it/s]
  1%|          | 104/10000 [00:45<32:11,  5.12it/s]
  1%|          | 105/10000 [00:45<32:06,  5.14it/s]
  1%|          | 106/10000 [00:45<31:50,  5.18it/s]
  1%|          | 107/10000 [00:45<32:31,  5.07it/s]
  1%|          | 108/10000 [00:45<34:59,  4.71it/s]
  1%|          | 109/10000 [00:46<35:01,  4.71it/s]
  1%|          | 110/10000 [00:46<33:49,  4.87it/s]
                                                   
{'loss': 1.1179, 'grad_norm': 0.3924192488193512, 'learning_rate': 2.18e-05}

  1%|          | 110/10000 [00:46<33:49,  4.87it/s]
  1%|          | 111/10000 [00:46<33:31,  4.92it/s]
  1%|          | 112/10000 [00:46<32:28,  5.07it/s]
  1%|          | 113/10000 [00:46<31:58,  5.15it/s]
  1%|          | 114/10000 [00:47<32:13,  5.11it/s]
  1%|          | 115/10000 [00:47<32:36,  5.05it/s]
  1%|          | 116/10000 [00:47<32:53,  5.01it/s]
  1%|          | 117/10000 [00:47<33:25,  4.93it/s]
  1%|          | 118/10000 [00:47<33:12,  4.96it/s]
  1%|          | 119/10000 [00:48<32:59,  4.99it/s]
  1%|          | 120/10000 [00:48<33:23,  4.93it/s]
                                                   
{'loss': 1.1116, 'grad_norm': 0.5948011875152588, 'learning_rate': 2.38e-05}

  1%|          | 120/10000 [00:48<33:23,  4.93it/s]
  1%|          | 121/10000 [00:48<33:17,  4.95it/s]
  1%|          | 122/10000 [00:48<33:34,  4.90it/s]
  1%|          | 123/10000 [00:48<33:55,  4.85it/s]
  1%|          | 124/10000 [00:49<33:03,  4.98it/s]
  1%|▏         | 125/10000 [00:49<32:36,  5.05it/s]
  1%|▏         | 126/10000 [00:49<32:52,  5.00it/s]
  1%|▏         | 127/10000 [00:49<32:16,  5.10it/s]
  1%|▏         | 128/10000 [00:49<32:19,  5.09it/s]
  1%|▏         | 129/10000 [00:50<32:31,  5.06it/s]
  1%|▏         | 130/10000 [00:50<32:38,  5.04it/s]
                                                   
{'loss': 1.1039, 'grad_norm': 0.3768727779388428, 'learning_rate': 2.58e-05}

  1%|▏         | 130/10000 [00:50<32:38,  5.04it/s]
  1%|▏         | 131/10000 [00:50<33:05,  4.97it/s]
  1%|▏         | 132/10000 [00:50<32:21,  5.08it/s]
  1%|▏         | 133/10000 [00:50<31:32,  5.21it/s]
  1%|▏         | 134/10000 [00:51<31:33,  5.21it/s]
  1%|▏         | 135/10000 [00:51<30:56,  5.31it/s]
  1%|▏         | 136/10000 [00:51<31:38,  5.20it/s]
  1%|▏         | 137/10000 [00:51<31:44,  5.18it/s]
  1%|▏         | 138/10000 [00:51<31:47,  5.17it/s]
  1%|▏         | 139/10000 [00:52<31:52,  5.16it/s]
  1%|▏         | 140/10000 [00:52<32:12,  5.10it/s]
                                                   
{'loss': 1.098, 'grad_norm': 0.24624425172805786, 'learning_rate': 2.7800000000000005e-05}

  1%|▏         | 140/10000 [00:52<32:12,  5.10it/s]
  1%|▏         | 141/10000 [00:52<32:36,  5.04it/s]
  1%|▏         | 142/10000 [00:52<31:58,  5.14it/s]
  1%|▏         | 143/10000 [00:52<32:03,  5.13it/s]
  1%|▏         | 144/10000 [00:53<31:59,  5.13it/s]
  1%|▏         | 145/10000 [00:53<31:41,  5.18it/s]
  1%|▏         | 146/10000 [00:53<31:52,  5.15it/s]
  1%|▏         | 147/10000 [00:53<31:40,  5.18it/s]
  1%|▏         | 148/10000 [00:53<31:01,  5.29it/s]
  1%|▏         | 149/10000 [00:54<31:07,  5.28it/s]
  2%|▏         | 150/10000 [00:54<31:27,  5.22it/s]
                                                   
{'loss': 1.1183, 'grad_norm': 0.3038891851902008, 'learning_rate': 2.98e-05}

  2%|▏         | 150/10000 [00:54<31:27,  5.22it/s]
  2%|▏         | 151/10000 [00:54<32:09,  5.10it/s]
  2%|▏         | 152/10000 [00:54<32:09,  5.10it/s]
  2%|▏         | 153/10000 [00:54<31:52,  5.15it/s]
  2%|▏         | 154/10000 [00:54<31:06,  5.28it/s]
  2%|▏         | 155/10000 [00:55<30:36,  5.36it/s]
  2%|▏         | 156/10000 [00:55<30:08,  5.44it/s]
  2%|▏         | 157/10000 [00:55<30:05,  5.45it/s]
  2%|▏         | 158/10000 [00:55<30:21,  5.40it/s]
  2%|▏         | 159/10000 [00:55<30:20,  5.41it/s]
  2%|▏         | 160/10000 [00:56<30:40,  5.35it/s]
                                                   
{'loss': 1.1043, 'grad_norm': 0.5524261593818665, 'learning_rate': 3.18e-05}

  2%|▏         | 160/10000 [00:56<30:40,  5.35it/s]
  2%|▏         | 161/10000 [00:56<31:16,  5.24it/s]
  2%|▏         | 162/10000 [00:56<31:19,  5.23it/s]
  2%|▏         | 163/10000 [00:56<30:53,  5.31it/s]
  2%|▏         | 164/10000 [00:56<30:26,  5.39it/s]
  2%|▏         | 165/10000 [00:57<29:57,  5.47it/s]
  2%|▏         | 166/10000 [00:57<29:48,  5.50it/s]
  2%|▏         | 167/10000 [00:57<29:36,  5.53it/s]
  2%|▏         | 168/10000 [00:57<29:31,  5.55it/s]
  2%|▏         | 169/10000 [00:57<29:19,  5.59it/s]
  2%|▏         | 170/10000 [00:57<29:25,  5.57it/s]
                                                   
{'loss': 1.1116, 'grad_norm': 0.4240923821926117, 'learning_rate': 3.38e-05}

  2%|▏         | 170/10000 [00:57<29:25,  5.57it/s]
  2%|▏         | 171/10000 [00:58<30:37,  5.35it/s]
  2%|▏         | 172/10000 [00:58<30:41,  5.34it/s]
  2%|▏         | 173/10000 [00:58<30:43,  5.33it/s]
  2%|▏         | 174/10000 [00:58<30:41,  5.33it/s]
  2%|▏         | 175/10000 [00:58<30:30,  5.37it/s]
  2%|▏         | 176/10000 [00:59<30:12,  5.42it/s]
  2%|▏         | 177/10000 [00:59<29:48,  5.49it/s]
  2%|▏         | 178/10000 [00:59<29:36,  5.53it/s]
  2%|▏         | 179/10000 [00:59<29:27,  5.56it/s]
  2%|▏         | 180/10000 [00:59<29:03,  5.63it/s]
                                                   
{'loss': 1.1087, 'grad_norm': 0.3693060874938965, 'learning_rate': 3.58e-05}

  2%|▏         | 180/10000 [00:59<29:03,  5.63it/s]
  2%|▏         | 181/10000 [00:59<29:34,  5.53it/s]
  2%|▏         | 182/10000 [01:00<29:58,  5.46it/s]
  2%|▏         | 183/10000 [01:00<29:37,  5.52it/s]
  2%|▏         | 184/10000 [01:00<29:37,  5.52it/s]
  2%|▏         | 185/10000 [01:00<29:26,  5.55it/s]
  2%|▏         | 186/10000 [01:00<29:22,  5.57it/s]
  2%|▏         | 187/10000 [01:01<29:22,  5.57it/s]
  2%|▏         | 188/10000 [01:01<29:59,  5.45it/s]
  2%|▏         | 189/10000 [01:01<32:21,  5.05it/s]
  2%|▏         | 190/10000 [01:01<33:55,  4.82it/s]
                                                   
{'loss': 1.0929, 'grad_norm': 0.7437363266944885, 'learning_rate': 3.7800000000000004e-05}

  2%|▏         | 190/10000 [01:01<33:55,  4.82it/s]
  2%|▏         | 191/10000 [01:01<35:30,  4.60it/s]
  2%|▏         | 192/10000 [01:02<35:46,  4.57it/s]
  2%|▏         | 193/10000 [01:02<36:03,  4.53it/s]
  2%|▏         | 194/10000 [01:02<33:45,  4.84it/s]
  2%|▏         | 195/10000 [01:02<32:15,  5.07it/s]
  2%|▏         | 196/10000 [01:02<31:18,  5.22it/s]
  2%|▏         | 197/10000 [01:03<30:47,  5.31it/s]
  2%|▏         | 198/10000 [01:03<31:54,  5.12it/s]
  2%|▏         | 199/10000 [01:03<34:28,  4.74it/s]
  2%|▏         | 200/10000 [01:03<34:05,  4.79it/s]
                                                   
{'loss': 1.0779, 'grad_norm': 0.5727037787437439, 'learning_rate': 3.9800000000000005e-05}

  2%|▏         | 200/10000 [01:03<34:05,  4.79it/s]
  2%|▏         | 201/10000 [01:03<34:42,  4.70it/s]
  2%|▏         | 202/10000 [01:04<34:24,  4.75it/s]
  2%|▏         | 203/10000 [01:04<33:28,  4.88it/s]
  2%|▏         | 204/10000 [01:04<32:59,  4.95it/s]
  2%|▏         | 205/10000 [01:04<34:04,  4.79it/s]
  2%|▏         | 206/10000 [01:04<33:09,  4.92it/s]
  2%|▏         | 207/10000 [01:05<32:42,  4.99it/s]
  2%|▏         | 208/10000 [01:05<33:26,  4.88it/s]
  2%|▏         | 209/10000 [01:05<34:59,  4.66it/s]
  2%|▏         | 210/10000 [01:05<35:17,  4.62it/s]
                                                   
{'loss': 1.0419, 'grad_norm': 0.6432830095291138, 'learning_rate': 4.18e-05}

  2%|▏         | 210/10000 [01:05<35:17,  4.62it/s]
  2%|▏         | 211/10000 [01:06<36:07,  4.52it/s]
  2%|▏         | 212/10000 [01:06<35:00,  4.66it/s]
  2%|▏         | 213/10000 [01:06<34:03,  4.79it/s]
  2%|▏         | 214/10000 [01:06<33:05,  4.93it/s]
  2%|▏         | 215/10000 [01:06<32:46,  4.98it/s]
  2%|▏         | 216/10000 [01:07<32:16,  5.05it/s]
  2%|▏         | 217/10000 [01:07<32:50,  4.96it/s]
  2%|▏         | 218/10000 [01:07<34:40,  4.70it/s]
  2%|▏         | 219/10000 [01:07<35:25,  4.60it/s]
  2%|▏         | 220/10000 [01:07<35:49,  4.55it/s]
                                                   
{'loss': 1.0195, 'grad_norm': 0.7623288035392761, 'learning_rate': 4.38e-05}

  2%|▏         | 220/10000 [01:07<35:49,  4.55it/s]
  2%|▏         | 221/10000 [01:08<36:41,  4.44it/s]
  2%|▏         | 222/10000 [01:08<35:36,  4.58it/s]
  2%|▏         | 223/10000 [01:08<34:01,  4.79it/s]
  2%|▏         | 224/10000 [01:08<33:11,  4.91it/s]
  2%|▏         | 225/10000 [01:08<32:53,  4.95it/s]
  2%|▏         | 226/10000 [01:09<33:14,  4.90it/s]
  2%|▏         | 227/10000 [01:09<34:10,  4.77it/s]
  2%|▏         | 228/10000 [01:09<35:13,  4.62it/s]
  2%|▏         | 229/10000 [01:09<36:07,  4.51it/s]
  2%|▏         | 230/10000 [01:10<36:15,  4.49it/s]
                                                   
{'loss': 1.0032, 'grad_norm': 0.9106892347335815, 'learning_rate': 4.58e-05}

  2%|▏         | 230/10000 [01:10<36:15,  4.49it/s]
  2%|▏         | 231/10000 [01:10<35:07,  4.64it/s]
  2%|▏         | 232/10000 [01:10<33:57,  4.79it/s]
  2%|▏         | 233/10000 [01:10<33:28,  4.86it/s]
  2%|▏         | 234/10000 [01:10<33:38,  4.84it/s]
  2%|▏         | 235/10000 [01:11<36:19,  4.48it/s]
  2%|▏         | 236/10000 [01:11<38:27,  4.23it/s]
  2%|▏         | 237/10000 [01:11<36:57,  4.40it/s]
  2%|▏         | 238/10000 [01:11<35:59,  4.52it/s]
  2%|▏         | 239/10000 [01:12<34:45,  4.68it/s]
  2%|▏         | 240/10000 [01:12<33:51,  4.80it/s]
                                                   
{'loss': 0.9891, 'grad_norm': 0.6734551787376404, 'learning_rate': 4.78e-05}

  2%|▏         | 240/10000 [01:12<33:51,  4.80it/s]
  2%|▏         | 241/10000 [01:12<34:24,  4.73it/s]
  2%|▏         | 242/10000 [01:12<33:52,  4.80it/s]
  2%|▏         | 243/10000 [01:12<35:04,  4.64it/s]
  2%|▏         | 244/10000 [01:13<34:43,  4.68it/s]
  2%|▏         | 245/10000 [01:13<35:19,  4.60it/s]
  2%|▏         | 246/10000 [01:13<35:13,  4.61it/s]
  2%|▏         | 247/10000 [01:13<35:33,  4.57it/s]
  2%|▏         | 248/10000 [01:13<35:34,  4.57it/s]
  2%|▏         | 249/10000 [01:14<35:02,  4.64it/s]
  2%|▎         | 250/10000 [01:14<34:47,  4.67it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 3: Caching shard...

Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.62 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.54 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 18.60 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 18.95 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.03 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.07 seconds
Rank 0, Worker 0: Caching shard...
[held-out-eval] step 250: eval/loss = 0.9484 (40s)

                                                   
{'loss': 0.9565, 'grad_norm': 0.8862823843955994, 'learning_rate': 4.9800000000000004e-05}

  2%|▎         | 250/10000 [01:54<34:47,  4.67it/s]
  3%|▎         | 251/10000 [01:54<32:50:56, 12.13s/it]
  3%|▎         | 252/10000 [01:54<23:09:16,  8.55s/it]
  3%|▎         | 253/10000 [01:54<16:22:50,  6.05s/it]
  3%|▎         | 254/10000 [01:54<11:37:35,  4.29s/it]
  3%|▎         | 255/10000 [01:55<8:18:33,  3.07s/it] 
  3%|▎         | 256/10000 [01:55<5:59:48,  2.22s/it]
  3%|▎         | 257/10000 [01:55<4:22:47,  1.62s/it]
  3%|▎         | 258/10000 [01:55<3:14:39,  1.20s/it]
  3%|▎         | 259/10000 [01:56<2:26:52,  1.11it/s]
  3%|▎         | 260/10000 [01:56<1:52:20,  1.45it/s]
                                                     
{'loss': 0.9401, 'grad_norm': 0.8855322003364563, 'learning_rate': 5.1800000000000005e-05}

  3%|▎         | 260/10000 [01:56<1:52:20,  1.45it/s]
  3%|▎         | 261/10000 [01:56<1:28:26,  1.84it/s]
  3%|▎         | 262/10000 [01:56<1:10:58,  2.29it/s]
  3%|▎         | 263/10000 [01:56<59:56,  2.71it/s]  
  3%|▎         | 264/10000 [01:57<52:05,  3.11it/s]
  3%|▎         | 265/10000 [01:57<47:21,  3.43it/s]
  3%|▎         | 266/10000 [01:57<43:45,  3.71it/s]
  3%|▎         | 267/10000 [01:57<41:21,  3.92it/s]
  3%|▎         | 268/10000 [01:57<39:33,  4.10it/s]
  3%|▎         | 269/10000 [01:58<37:20,  4.34it/s]
  3%|▎         | 270/10000 [01:58<35:43,  4.54it/s]
                                                   
{'loss': 0.9177, 'grad_norm': 0.6994087100028992, 'learning_rate': 5.380000000000001e-05}

  3%|▎         | 270/10000 [01:58<35:43,  4.54it/s]
  3%|▎         | 271/10000 [01:58<34:31,  4.70it/s]
  3%|▎         | 272/10000 [01:58<33:38,  4.82it/s]
  3%|▎         | 273/10000 [01:58<34:16,  4.73it/s]
  3%|▎         | 274/10000 [01:59<36:49,  4.40it/s]
  3%|▎         | 275/10000 [01:59<37:30,  4.32it/s]
  3%|▎         | 276/10000 [01:59<37:54,  4.27it/s]
  3%|▎         | 277/10000 [01:59<36:21,  4.46it/s]
  3%|▎         | 278/10000 [02:00<34:55,  4.64it/s]
  3%|▎         | 279/10000 [02:00<34:08,  4.75it/s]
  3%|▎         | 280/10000 [02:00<33:33,  4.83it/s]
                                                   
{'loss': 0.8871, 'grad_norm': 0.8596734404563904, 'learning_rate': 5.580000000000001e-05}

  3%|▎         | 280/10000 [02:00<33:33,  4.83it/s]
  3%|▎         | 281/10000 [02:00<34:17,  4.72it/s]
  3%|▎         | 282/10000 [02:00<35:12,  4.60it/s]
  3%|▎         | 283/10000 [02:01<35:12,  4.60it/s]
  3%|▎         | 284/10000 [02:01<36:13,  4.47it/s]
  3%|▎         | 285/10000 [02:01<38:26,  4.21it/s]
  3%|▎         | 286/10000 [02:01<38:08,  4.25it/s]
  3%|▎         | 287/10000 [02:02<38:11,  4.24it/s]
  3%|▎         | 288/10000 [02:02<36:03,  4.49it/s]
  3%|▎         | 289/10000 [02:02<35:14,  4.59it/s]
  3%|▎         | 290/10000 [02:02<38:21,  4.22it/s]
                                                   
{'loss': 0.8865, 'grad_norm': 0.7085843682289124, 'learning_rate': 5.7799999999999995e-05}

  3%|▎         | 290/10000 [02:02<38:21,  4.22it/s]
  3%|▎         | 291/10000 [02:02<37:35,  4.30it/s]
  3%|▎         | 292/10000 [02:03<37:32,  4.31it/s]
  3%|▎         | 293/10000 [02:03<38:03,  4.25it/s]
  3%|▎         | 294/10000 [02:03<38:03,  4.25it/s]
  3%|▎         | 295/10000 [02:03<36:29,  4.43it/s]
  3%|▎         | 296/10000 [02:04<35:19,  4.58it/s]
  3%|▎         | 297/10000 [02:04<34:15,  4.72it/s]
  3%|▎         | 298/10000 [02:04<33:09,  4.88it/s]
  3%|▎         | 299/10000 [02:04<32:46,  4.93it/s]
  3%|▎         | 300/10000 [02:04<33:10,  4.87it/s]
                                                   
{'loss': 0.8617, 'grad_norm': 0.7338278889656067, 'learning_rate': 5.9800000000000003e-05}

  3%|▎         | 300/10000 [02:04<33:10,  4.87it/s]
  3%|▎         | 301/10000 [02:05<34:08,  4.74it/s]
  3%|▎         | 302/10000 [02:05<34:42,  4.66it/s]
  3%|▎         | 303/10000 [02:05<37:00,  4.37it/s]
  3%|▎         | 304/10000 [02:05<36:19,  4.45it/s]
  3%|▎         | 305/10000 [02:06<35:06,  4.60it/s]
  3%|▎         | 306/10000 [02:06<34:16,  4.71it/s]
  3%|▎         | 307/10000 [02:06<33:28,  4.83it/s]
  3%|▎         | 308/10000 [02:06<32:32,  4.96it/s]
  3%|▎         | 309/10000 [02:06<32:41,  4.94it/s]
  3%|▎         | 310/10000 [02:07<32:38,  4.95it/s]
                                                   
{'loss': 0.8596, 'grad_norm': 1.1336745023727417, 'learning_rate': 6.18e-05}

  3%|▎         | 310/10000 [02:07<32:38,  4.95it/s]
  3%|▎         | 311/10000 [02:07<33:38,  4.80it/s]
  3%|▎         | 312/10000 [02:07<33:38,  4.80it/s]
  3%|▎         | 313/10000 [02:07<33:56,  4.76it/s]
  3%|▎         | 314/10000 [02:07<34:22,  4.70it/s]
  3%|▎         | 315/10000 [02:08<34:24,  4.69it/s]
  3%|▎         | 316/10000 [02:08<34:04,  4.74it/s]
  3%|▎         | 317/10000 [02:08<33:35,  4.81it/s]
  3%|▎         | 318/10000 [02:08<33:12,  4.86it/s]
  3%|▎         | 319/10000 [02:08<33:07,  4.87it/s]
  3%|▎         | 320/10000 [02:09<32:55,  4.90it/s]
                                                   
{'loss': 0.8446, 'grad_norm': 0.7997697591781616, 'learning_rate': 6.38e-05}

  3%|▎         | 320/10000 [02:09<32:55,  4.90it/s]
  3%|▎         | 321/10000 [02:09<34:50,  4.63it/s]
  3%|▎         | 322/10000 [02:09<35:40,  4.52it/s]
  3%|▎         | 323/10000 [02:09<35:00,  4.61it/s]
  3%|▎         | 324/10000 [02:09<34:11,  4.72it/s]
  3%|▎         | 325/10000 [02:10<33:35,  4.80it/s]
  3%|▎         | 326/10000 [02:10<33:17,  4.84it/s]
  3%|▎         | 327/10000 [02:10<33:11,  4.86it/s]
  3%|▎         | 328/10000 [02:10<33:07,  4.87it/s]
  3%|▎         | 329/10000 [02:11<33:47,  4.77it/s]
  3%|▎         | 330/10000 [02:11<34:21,  4.69it/s]
                                                   
{'loss': 0.837, 'grad_norm': 0.797099232673645, 'learning_rate': 6.58e-05}

  3%|▎         | 330/10000 [02:11<34:21,  4.69it/s]
  3%|▎         | 331/10000 [02:11<34:27,  4.68it/s]
  3%|▎         | 332/10000 [02:11<34:35,  4.66it/s]
  3%|▎         | 333/10000 [02:11<34:50,  4.62it/s]
  3%|▎         | 334/10000 [02:12<34:56,  4.61it/s]
  3%|▎         | 335/10000 [02:12<34:04,  4.73it/s]
  3%|▎         | 336/10000 [02:12<33:22,  4.83it/s]
  3%|▎         | 337/10000 [02:12<32:48,  4.91it/s]
  3%|▎         | 338/10000 [02:12<32:11,  5.00it/s]
  3%|▎         | 339/10000 [02:13<32:08,  5.01it/s]
  3%|▎         | 340/10000 [02:13<32:07,  5.01it/s]
                                                   
{'loss': 0.8112, 'grad_norm': 0.9630879759788513, 'learning_rate': 6.780000000000001e-05}

  3%|▎         | 340/10000 [02:13<32:07,  5.01it/s]
  3%|▎         | 341/10000 [02:13<33:09,  4.85it/s]
  3%|▎         | 342/10000 [02:13<32:55,  4.89it/s]
  3%|▎         | 343/10000 [02:13<32:26,  4.96it/s]
  3%|▎         | 344/10000 [02:14<31:55,  5.04it/s]
  3%|▎         | 345/10000 [02:14<31:32,  5.10it/s]
  3%|▎         | 346/10000 [02:14<31:10,  5.16it/s]
  3%|▎         | 347/10000 [02:14<31:19,  5.13it/s]
  3%|▎         | 348/10000 [02:14<31:24,  5.12it/s]
  3%|▎         | 349/10000 [02:15<31:05,  5.17it/s]
  4%|▎         | 350/10000 [02:15<30:46,  5.23it/s]
                                                   
{'loss': 0.8131, 'grad_norm': 1.303208589553833, 'learning_rate': 6.98e-05}

  4%|▎         | 350/10000 [02:15<30:46,  5.23it/s]
  4%|▎         | 351/10000 [02:15<32:23,  4.96it/s]
  4%|▎         | 352/10000 [02:15<32:04,  5.01it/s]
  4%|▎         | 353/10000 [02:15<33:07,  4.85it/s]
  4%|▎         | 354/10000 [02:16<33:07,  4.85it/s]
  4%|▎         | 355/10000 [02:16<33:14,  4.84it/s]
  4%|▎         | 356/10000 [02:16<31:33,  5.09it/s]
  4%|▎         | 357/10000 [02:16<31:19,  5.13it/s]
  4%|▎         | 358/10000 [02:16<32:03,  5.01it/s]
  4%|▎         | 359/10000 [02:17<32:13,  4.99it/s]
  4%|▎         | 360/10000 [02:17<33:26,  4.80it/s]
                                                   
{'loss': 0.8073, 'grad_norm': 1.0860744714736938, 'learning_rate': 7.18e-05}

  4%|▎         | 360/10000 [02:17<33:26,  4.80it/s]
  4%|▎         | 361/10000 [02:17<35:10,  4.57it/s]
  4%|▎         | 362/10000 [02:17<35:13,  4.56it/s]
  4%|▎         | 363/10000 [02:17<34:21,  4.68it/s]
  4%|▎         | 364/10000 [02:18<32:46,  4.90it/s]
  4%|▎         | 365/10000 [02:18<31:42,  5.06it/s]
  4%|▎         | 366/10000 [02:18<31:19,  5.12it/s]
  4%|▎         | 367/10000 [02:18<31:49,  5.04it/s]
  4%|▎         | 368/10000 [02:18<32:13,  4.98it/s]
  4%|▎         | 369/10000 [02:19<31:58,  5.02it/s]
  4%|▎         | 370/10000 [02:19<31:53,  5.03it/s]
                                                   
{'loss': 0.8007, 'grad_norm': 0.9043018221855164, 'learning_rate': 7.38e-05}

  4%|▎         | 370/10000 [02:19<31:53,  5.03it/s]
  4%|▎         | 371/10000 [02:19<32:36,  4.92it/s]
  4%|▎         | 372/10000 [02:19<31:49,  5.04it/s]
  4%|▎         | 373/10000 [02:19<31:20,  5.12it/s]
  4%|▎         | 374/10000 [02:20<30:54,  5.19it/s]
  4%|▍         | 375/10000 [02:20<31:23,  5.11it/s]
  4%|▍         | 376/10000 [02:20<31:37,  5.07it/s]
  4%|▍         | 377/10000 [02:20<31:35,  5.08it/s]
  4%|▍         | 378/10000 [02:20<31:42,  5.06it/s]
  4%|▍         | 379/10000 [02:21<31:35,  5.08it/s]
  4%|▍         | 380/10000 [02:21<31:10,  5.14it/s]
                                                   
{'loss': 0.793, 'grad_norm': 0.8608036041259766, 'learning_rate': 7.58e-05}

  4%|▍         | 380/10000 [02:21<31:10,  5.14it/s]
  4%|▍         | 381/10000 [02:21<31:01,  5.17it/s]
  4%|▍         | 382/10000 [02:21<30:35,  5.24it/s]
  4%|▍         | 383/10000 [02:21<30:04,  5.33it/s]
  4%|▍         | 384/10000 [02:22<29:42,  5.40it/s]
  4%|▍         | 385/10000 [02:22<30:01,  5.34it/s]
  4%|▍         | 386/10000 [02:22<30:00,  5.34it/s]
  4%|▍         | 387/10000 [02:22<30:15,  5.30it/s]
  4%|▍         | 388/10000 [02:22<30:36,  5.23it/s]
  4%|▍         | 389/10000 [02:22<30:10,  5.31it/s]
  4%|▍         | 390/10000 [02:23<30:13,  5.30it/s]
                                                   
{'loss': 0.7669, 'grad_norm': 1.2061337232589722, 'learning_rate': 7.780000000000001e-05}

  4%|▍         | 390/10000 [02:23<30:13,  5.30it/s]
  4%|▍         | 391/10000 [02:23<30:32,  5.24it/s]
  4%|▍         | 392/10000 [02:23<30:03,  5.33it/s]
  4%|▍         | 393/10000 [02:23<30:03,  5.33it/s]
  4%|▍         | 394/10000 [02:23<30:37,  5.23it/s]
  4%|▍         | 395/10000 [02:24<30:59,  5.16it/s]
  4%|▍         | 396/10000 [02:24<31:05,  5.15it/s]
  4%|▍         | 397/10000 [02:24<31:20,  5.11it/s]
  4%|▍         | 398/10000 [02:24<31:08,  5.14it/s]
  4%|▍         | 399/10000 [02:24<30:42,  5.21it/s]
  4%|▍         | 400/10000 [02:25<30:24,  5.26it/s]
                                                   
{'loss': 0.7702, 'grad_norm': 0.7855536341667175, 'learning_rate': 7.98e-05}

  4%|▍         | 400/10000 [02:25<30:24,  5.26it/s]
  4%|▍         | 401/10000 [02:25<30:32,  5.24it/s]
  4%|▍         | 402/10000 [02:25<30:43,  5.21it/s]
  4%|▍         | 403/10000 [02:25<31:27,  5.08it/s]
  4%|▍         | 404/10000 [02:25<31:12,  5.12it/s]
  4%|▍         | 405/10000 [02:26<31:12,  5.13it/s]
  4%|▍         | 406/10000 [02:26<31:50,  5.02it/s]
  4%|▍         | 407/10000 [02:26<31:23,  5.09it/s]
  4%|▍         | 408/10000 [02:26<31:30,  5.07it/s]
  4%|▍         | 409/10000 [02:26<31:25,  5.09it/s]
  4%|▍         | 410/10000 [02:27<31:25,  5.09it/s]
                                                   
{'loss': 0.7713, 'grad_norm': 0.7040292620658875, 'learning_rate': 8.18e-05}

  4%|▍         | 410/10000 [02:27<31:25,  5.09it/s]
  4%|▍         | 411/10000 [02:27<31:18,  5.10it/s]
  4%|▍         | 412/10000 [02:27<30:41,  5.21it/s]
  4%|▍         | 413/10000 [02:27<30:06,  5.31it/s]
  4%|▍         | 414/10000 [02:27<30:02,  5.32it/s]
  4%|▍         | 415/10000 [02:28<30:37,  5.22it/s]
  4%|▍         | 416/10000 [02:28<31:54,  5.01it/s]
  4%|▍         | 417/10000 [02:28<32:42,  4.88it/s]
  4%|▍         | 418/10000 [02:28<32:35,  4.90it/s]
  4%|▍         | 419/10000 [02:28<32:18,  4.94it/s]
  4%|▍         | 420/10000 [02:29<31:20,  5.10it/s]
                                                   
{'loss': 0.7545, 'grad_norm': 0.7931767106056213, 'learning_rate': 8.38e-05}

  4%|▍         | 420/10000 [02:29<31:20,  5.10it/s]
  4%|▍         | 421/10000 [02:29<30:54,  5.16it/s]
  4%|▍         | 422/10000 [02:29<30:10,  5.29it/s]
  4%|▍         | 423/10000 [02:29<30:55,  5.16it/s]
  4%|▍         | 424/10000 [02:29<30:48,  5.18it/s]
  4%|▍         | 425/10000 [02:29<30:46,  5.18it/s]
  4%|▍         | 426/10000 [02:30<30:43,  5.19it/s]
  4%|▍         | 427/10000 [02:30<30:37,  5.21it/s]
  4%|▍         | 428/10000 [02:30<30:26,  5.24it/s]
  4%|▍         | 429/10000 [02:30<30:15,  5.27it/s]
  4%|▍         | 430/10000 [02:30<30:01,  5.31it/s]
                                                   
{'loss': 0.7419, 'grad_norm': 0.7801165580749512, 'learning_rate': 8.58e-05}

  4%|▍         | 430/10000 [02:30<30:01,  5.31it/s]
  4%|▍         | 431/10000 [02:31<30:23,  5.25it/s]
  4%|▍         | 432/10000 [02:31<30:35,  5.21it/s]
  4%|▍         | 433/10000 [02:31<31:00,  5.14it/s]
  4%|▍         | 434/10000 [02:31<31:11,  5.11it/s]
  4%|▍         | 435/10000 [02:31<31:25,  5.07it/s]
  4%|▍         | 436/10000 [02:32<31:45,  5.02it/s]
  4%|▍         | 437/10000 [02:32<32:16,  4.94it/s]
  4%|▍         | 438/10000 [02:32<32:06,  4.96it/s]
  4%|▍         | 439/10000 [02:32<30:56,  5.15it/s]
  4%|▍         | 440/10000 [02:32<32:00,  4.98it/s]
                                                   
{'loss': 0.7437, 'grad_norm': 0.8844775557518005, 'learning_rate': 8.78e-05}

  4%|▍         | 440/10000 [02:32<32:00,  4.98it/s]
  4%|▍         | 441/10000 [02:33<32:45,  4.86it/s]
  4%|▍         | 442/10000 [02:33<34:50,  4.57it/s]
  4%|▍         | 443/10000 [02:33<37:02,  4.30it/s]
  4%|▍         | 444/10000 [02:33<37:20,  4.27it/s]
  4%|▍         | 445/10000 [02:34<37:41,  4.23it/s]
  4%|▍         | 446/10000 [02:34<37:17,  4.27it/s]
  4%|▍         | 447/10000 [02:34<35:32,  4.48it/s]
  4%|▍         | 448/10000 [02:34<33:46,  4.71it/s]
  4%|▍         | 449/10000 [02:34<33:01,  4.82it/s]
  4%|▍         | 450/10000 [02:35<32:33,  4.89it/s]
                                                   
{'loss': 0.719, 'grad_norm': 0.9408592581748962, 'learning_rate': 8.98e-05}

  4%|▍         | 450/10000 [02:35<32:33,  4.89it/s]
  5%|▍         | 451/10000 [02:35<33:48,  4.71it/s]
  5%|▍         | 452/10000 [02:35<34:51,  4.57it/s]
  5%|▍         | 453/10000 [02:35<36:52,  4.31it/s]
  5%|▍         | 454/10000 [02:36<37:43,  4.22it/s]
  5%|▍         | 455/10000 [02:36<36:38,  4.34it/s]
  5%|▍         | 456/10000 [02:36<35:31,  4.48it/s]
  5%|▍         | 457/10000 [02:36<34:00,  4.68it/s]
  5%|▍         | 458/10000 [02:36<32:40,  4.87it/s]
  5%|▍         | 459/10000 [02:37<32:35,  4.88it/s]
  5%|▍         | 460/10000 [02:37<32:37,  4.87it/s]
                                                   
{'loss': 0.7393, 'grad_norm': 1.0986884832382202, 'learning_rate': 9.180000000000001e-05}

  5%|▍         | 460/10000 [02:37<32:37,  4.87it/s]
  5%|▍         | 461/10000 [02:37<33:17,  4.78it/s]
  5%|▍         | 462/10000 [02:37<35:12,  4.51it/s]
  5%|▍         | 463/10000 [02:38<37:29,  4.24it/s]
  5%|▍         | 464/10000 [02:38<38:27,  4.13it/s]
  5%|▍         | 465/10000 [02:38<38:35,  4.12it/s]
  5%|▍         | 466/10000 [02:38<37:41,  4.22it/s]
  5%|▍         | 467/10000 [02:38<35:37,  4.46it/s]
  5%|▍         | 468/10000 [02:39<33:49,  4.70it/s]
  5%|▍         | 469/10000 [02:39<33:17,  4.77it/s]
  5%|▍         | 470/10000 [02:39<33:28,  4.74it/s]
                                                   
{'loss': 0.7328, 'grad_norm': 0.7132677435874939, 'learning_rate': 9.38e-05}

  5%|▍         | 470/10000 [02:39<33:28,  4.74it/s]
  5%|▍         | 471/10000 [02:39<34:54,  4.55it/s]
  5%|▍         | 472/10000 [02:40<36:00,  4.41it/s]
  5%|▍         | 473/10000 [02:40<37:23,  4.25it/s]
  5%|▍         | 474/10000 [02:40<38:01,  4.18it/s]
  5%|▍         | 475/10000 [02:40<38:38,  4.11it/s]
  5%|▍         | 476/10000 [02:41<37:33,  4.23it/s]
  5%|▍         | 477/10000 [02:41<35:39,  4.45it/s]
  5%|▍         | 478/10000 [02:41<34:50,  4.55it/s]
  5%|▍         | 479/10000 [02:41<34:24,  4.61it/s]
  5%|▍         | 480/10000 [02:41<35:07,  4.52it/s]
                                                   
{'loss': 0.7285, 'grad_norm': 0.8457934856414795, 'learning_rate': 9.58e-05}

  5%|▍         | 480/10000 [02:41<35:07,  4.52it/s]
  5%|▍         | 481/10000 [02:42<38:00,  4.17it/s]
  5%|▍         | 482/10000 [02:42<38:33,  4.11it/s]
  5%|▍         | 483/10000 [02:42<38:37,  4.11it/s]
  5%|▍         | 484/10000 [02:42<37:29,  4.23it/s]
  5%|▍         | 485/10000 [02:43<35:17,  4.49it/s]
  5%|▍         | 486/10000 [02:43<34:05,  4.65it/s]
  5%|▍         | 487/10000 [02:43<33:26,  4.74it/s]
  5%|▍         | 488/10000 [02:43<33:13,  4.77it/s]
  5%|▍         | 489/10000 [02:43<33:58,  4.67it/s]
  5%|▍         | 490/10000 [02:44<36:09,  4.38it/s]
                                                   
{'loss': 0.6974, 'grad_norm': 0.8056934475898743, 'learning_rate': 9.78e-05}

  5%|▍         | 490/10000 [02:44<36:09,  4.38it/s]
  5%|▍         | 491/10000 [02:44<36:50,  4.30it/s]
  5%|▍         | 492/10000 [02:44<36:54,  4.29it/s]
  5%|▍         | 493/10000 [02:44<35:25,  4.47it/s]
  5%|▍         | 494/10000 [02:45<34:09,  4.64it/s]
  5%|▍         | 495/10000 [02:45<33:32,  4.72it/s]
  5%|▍         | 496/10000 [02:45<33:24,  4.74it/s]
  5%|▍         | 497/10000 [02:45<33:28,  4.73it/s]
  5%|▍         | 498/10000 [02:45<35:08,  4.51it/s]
  5%|▍         | 499/10000 [02:46<37:33,  4.22it/s]
  5%|▌         | 500/10000 [02:46<38:20,  4.13it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 19.44 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 20.99 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 21.06 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 21.12 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 21.51 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 21.85 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 500: eval/loss = 0.7051 (42s)

                                                   
{'loss': 0.7045, 'grad_norm': 0.7358251214027405, 'learning_rate': 9.98e-05}

  5%|▌         | 500/10000 [03:28<38:20,  4.13it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-500/wandb_config.json

  5%|▌         | 501/10000 [03:57<57:04:06, 21.63s/it]
  5%|▌         | 502/10000 [03:58<40:05:01, 15.19s/it]
  5%|▌         | 503/10000 [03:58<28:11:41, 10.69s/it]
  5%|▌         | 504/10000 [03:58<19:53:53,  7.54s/it]
  5%|▌         | 505/10000 [03:58<14:04:05,  5.33s/it]
  5%|▌         | 506/10000 [03:58<9:59:25,  3.79s/it] 
  5%|▌         | 507/10000 [03:59<7:09:16,  2.71s/it]
  5%|▌         | 508/10000 [03:59<5:08:57,  1.95s/it]
  5%|▌         | 509/10000 [03:59<3:44:58,  1.42s/it]
  5%|▌         | 510/10000 [03:59<2:48:21,  1.06s/it]
                                                     
{'loss': 0.7006, 'grad_norm': 0.7621918320655823, 'learning_rate': 9.9999778549206e-05}

  5%|▌         | 510/10000 [03:59<2:48:21,  1.06s/it]
  5%|▌         | 511/10000 [03:59<2:08:09,  1.23it/s]
  5%|▌         | 512/10000 [04:00<1:39:38,  1.59it/s]
  5%|▌         | 513/10000 [04:00<1:20:24,  1.97it/s]
  5%|▌         | 514/10000 [04:00<1:05:58,  2.40it/s]
  5%|▌         | 515/10000 [04:00<55:48,  2.83it/s]  
  5%|▌         | 516/10000 [04:00<48:49,  3.24it/s]
  5%|▌         | 517/10000 [04:01<44:07,  3.58it/s]
  5%|▌         | 518/10000 [04:01<40:33,  3.90it/s]
  5%|▌         | 519/10000 [04:01<38:13,  4.13it/s]
  5%|▌         | 520/10000 [04:01<36:35,  4.32it/s]
                                                   
{'loss': 0.7058, 'grad_norm': 0.7946559190750122, 'learning_rate': 9.999901304280685e-05}

  5%|▌         | 520/10000 [04:01<36:35,  4.32it/s]
  5%|▌         | 521/10000 [04:01<36:09,  4.37it/s]
  5%|▌         | 522/10000 [04:02<35:36,  4.44it/s]
  5%|▌         | 523/10000 [04:02<34:46,  4.54it/s]
  5%|▌         | 524/10000 [04:02<33:42,  4.68it/s]
  5%|▌         | 525/10000 [04:02<32:07,  4.92it/s]
  5%|▌         | 526/10000 [04:02<30:38,  5.15it/s]
  5%|▌         | 527/10000 [04:03<29:49,  5.29it/s]
  5%|▌         | 528/10000 [04:03<29:19,  5.38it/s]
  5%|▌         | 529/10000 [04:03<29:30,  5.35it/s]
  5%|▌         | 530/10000 [04:03<29:37,  5.33it/s]
                                                   
{'loss': 0.7028, 'grad_norm': 0.7282401323318481, 'learning_rate': 9.999770075521164e-05}

  5%|▌         | 530/10000 [04:03<29:37,  5.33it/s]
  5%|▌         | 531/10000 [04:03<30:54,  5.10it/s]
  5%|▌         | 532/10000 [04:04<31:08,  5.07it/s]
  5%|▌         | 533/10000 [04:04<31:07,  5.07it/s]
  5%|▌         | 534/10000 [04:04<31:05,  5.08it/s]
  5%|▌         | 535/10000 [04:04<30:01,  5.25it/s]
  5%|▌         | 536/10000 [04:04<29:29,  5.35it/s]
  5%|▌         | 537/10000 [04:05<29:03,  5.43it/s]
  5%|▌         | 538/10000 [04:05<28:30,  5.53it/s]
  5%|▌         | 539/10000 [04:05<28:27,  5.54it/s]
  5%|▌         | 540/10000 [04:05<28:53,  5.46it/s]
                                                   
{'loss': 0.6907, 'grad_norm': 0.8195558786392212, 'learning_rate': 9.99958417007713e-05}

  5%|▌         | 540/10000 [04:05<28:53,  5.46it/s]
  5%|▌         | 541/10000 [04:05<29:31,  5.34it/s]
  5%|▌         | 542/10000 [04:05<30:15,  5.21it/s]
  5%|▌         | 543/10000 [04:06<30:43,  5.13it/s]
  5%|▌         | 544/10000 [04:06<30:42,  5.13it/s]
  5%|▌         | 545/10000 [04:06<31:25,  5.02it/s]
  5%|▌         | 546/10000 [04:06<30:54,  5.10it/s]
  5%|▌         | 547/10000 [04:06<30:03,  5.24it/s]
  5%|▌         | 548/10000 [04:07<29:47,  5.29it/s]
  5%|▌         | 549/10000 [04:07<29:27,  5.35it/s]
  6%|▌         | 550/10000 [04:07<29:00,  5.43it/s]
                                                   
{'loss': 0.6736, 'grad_norm': 0.7697458863258362, 'learning_rate': 9.999343589981615e-05}

  6%|▌         | 550/10000 [04:07<29:00,  5.43it/s]
  6%|▌         | 551/10000 [04:07<31:00,  5.08it/s]
  6%|▌         | 552/10000 [04:07<32:33,  4.84it/s]
  6%|▌         | 553/10000 [04:08<34:13,  4.60it/s]
  6%|▌         | 554/10000 [04:08<33:20,  4.72it/s]
  6%|▌         | 555/10000 [04:08<31:58,  4.92it/s]
  6%|▌         | 556/10000 [04:08<30:46,  5.11it/s]
  6%|▌         | 557/10000 [04:08<29:57,  5.25it/s]
  6%|▌         | 558/10000 [04:09<29:34,  5.32it/s]
  6%|▌         | 559/10000 [04:09<30:06,  5.23it/s]
  6%|▌         | 560/10000 [04:09<30:48,  5.11it/s]
                                                   
{'loss': 0.6946, 'grad_norm': 0.8372205495834351, 'learning_rate': 9.999048337865568e-05}

  6%|▌         | 560/10000 [04:09<30:48,  5.11it/s]
  6%|▌         | 561/10000 [04:09<31:20,  5.02it/s]
  6%|▌         | 562/10000 [04:09<31:27,  5.00it/s]
  6%|▌         | 563/10000 [04:10<31:15,  5.03it/s]
  6%|▌         | 564/10000 [04:10<30:57,  5.08it/s]
  6%|▌         | 565/10000 [04:10<29:56,  5.25it/s]
  6%|▌         | 566/10000 [04:10<29:47,  5.28it/s]
  6%|▌         | 567/10000 [04:10<29:58,  5.25it/s]
  6%|▌         | 568/10000 [04:11<30:40,  5.12it/s]
  6%|▌         | 569/10000 [04:11<30:44,  5.11it/s]
  6%|▌         | 570/10000 [04:11<31:07,  5.05it/s]
                                                   
{'loss': 0.6724, 'grad_norm': 0.7512030601501465, 'learning_rate': 9.998698416957815e-05}

  6%|▌         | 570/10000 [04:11<31:07,  5.05it/s]
  6%|▌         | 571/10000 [04:11<31:42,  4.96it/s]
  6%|▌         | 572/10000 [04:11<30:33,  5.14it/s]
  6%|▌         | 573/10000 [04:12<30:13,  5.20it/s]
  6%|▌         | 574/10000 [04:12<30:09,  5.21it/s]
  6%|▌         | 575/10000 [04:12<29:17,  5.36it/s]
  6%|▌         | 576/10000 [04:12<29:53,  5.26it/s]
  6%|▌         | 577/10000 [04:12<31:04,  5.05it/s]
  6%|▌         | 578/10000 [04:13<33:22,  4.70it/s]
  6%|▌         | 579/10000 [04:13<32:11,  4.88it/s]
  6%|▌         | 580/10000 [04:13<31:24,  5.00it/s]
                                                   
{'loss': 0.6794, 'grad_norm': 0.6948479413986206, 'learning_rate': 9.998293831085037e-05}

  6%|▌         | 580/10000 [04:13<31:24,  5.00it/s]
  6%|▌         | 581/10000 [04:13<31:27,  4.99it/s]
  6%|▌         | 582/10000 [04:13<31:44,  4.95it/s]
  6%|▌         | 583/10000 [04:14<31:49,  4.93it/s]
  6%|▌         | 584/10000 [04:14<31:39,  4.96it/s]
  6%|▌         | 585/10000 [04:14<31:11,  5.03it/s]
  6%|▌         | 586/10000 [04:14<30:56,  5.07it/s]
  6%|▌         | 587/10000 [04:14<30:35,  5.13it/s]
  6%|▌         | 588/10000 [04:15<30:42,  5.11it/s]
  6%|▌         | 589/10000 [04:15<30:00,  5.23it/s]
  6%|▌         | 590/10000 [04:15<29:53,  5.25it/s]
                                                   
{'loss': 0.6794, 'grad_norm': 0.8155792951583862, 'learning_rate': 9.997834584671719e-05}

  6%|▌         | 590/10000 [04:15<29:53,  5.25it/s]
  6%|▌         | 591/10000 [04:15<30:13,  5.19it/s]
  6%|▌         | 592/10000 [04:15<29:56,  5.24it/s]
  6%|▌         | 593/10000 [04:16<30:07,  5.21it/s]
  6%|▌         | 594/10000 [04:16<30:00,  5.23it/s]
  6%|▌         | 595/10000 [04:16<29:28,  5.32it/s]
  6%|▌         | 596/10000 [04:16<29:28,  5.32it/s]
  6%|▌         | 597/10000 [04:16<29:27,  5.32it/s]
  6%|▌         | 598/10000 [04:16<29:38,  5.29it/s]
  6%|▌         | 599/10000 [04:17<29:32,  5.30it/s]
  6%|▌         | 600/10000 [04:17<29:39,  5.28it/s]
                                                   
{'loss': 0.6785, 'grad_norm': 0.7938534617424011, 'learning_rate': 9.997320682740107e-05}

  6%|▌         | 600/10000 [04:17<29:39,  5.28it/s]
  6%|▌         | 601/10000 [04:17<30:11,  5.19it/s]
  6%|▌         | 602/10000 [04:17<29:52,  5.24it/s]
  6%|▌         | 603/10000 [04:17<29:53,  5.24it/s]
  6%|▌         | 604/10000 [04:18<29:27,  5.32it/s]
  6%|▌         | 605/10000 [04:18<29:40,  5.28it/s]
  6%|▌         | 606/10000 [04:18<29:42,  5.27it/s]
  6%|▌         | 607/10000 [04:18<29:30,  5.31it/s]
  6%|▌         | 608/10000 [04:18<29:16,  5.35it/s]
  6%|▌         | 609/10000 [04:19<28:57,  5.41it/s]
  6%|▌         | 610/10000 [04:19<28:37,  5.47it/s]
                                                   
{'loss': 0.6651, 'grad_norm': 1.145626187324524, 'learning_rate': 9.996752130910149e-05}

  6%|▌         | 610/10000 [04:19<28:37,  5.47it/s]
  6%|▌         | 611/10000 [04:19<28:40,  5.46it/s]
  6%|▌         | 612/10000 [04:19<28:21,  5.52it/s]
  6%|▌         | 613/10000 [04:19<28:06,  5.57it/s]
  6%|▌         | 614/10000 [04:19<27:45,  5.63it/s]
  6%|▌         | 615/10000 [04:20<27:53,  5.61it/s]
  6%|▌         | 616/10000 [04:20<28:00,  5.58it/s]
  6%|▌         | 617/10000 [04:20<28:10,  5.55it/s]
  6%|▌         | 618/10000 [04:20<28:54,  5.41it/s]
  6%|▌         | 619/10000 [04:20<30:31,  5.12it/s]
  6%|▌         | 620/10000 [04:21<31:52,  4.90it/s]
                                                   
{'loss': 0.6574, 'grad_norm': 0.9207320213317871, 'learning_rate': 9.99612893539944e-05}

  6%|▌         | 620/10000 [04:21<31:52,  4.90it/s]
  6%|▌         | 621/10000 [04:21<34:14,  4.57it/s]
  6%|▌         | 622/10000 [04:21<35:37,  4.39it/s]
  6%|▌         | 623/10000 [04:21<35:55,  4.35it/s]
  6%|▌         | 624/10000 [04:22<33:50,  4.62it/s]
  6%|▋         | 625/10000 [04:22<32:40,  4.78it/s]
  6%|▋         | 626/10000 [04:22<32:01,  4.88it/s]
  6%|▋         | 627/10000 [04:22<31:23,  4.98it/s]
  6%|▋         | 628/10000 [04:22<31:56,  4.89it/s]
  6%|▋         | 629/10000 [04:23<32:19,  4.83it/s]
  6%|▋         | 630/10000 [04:23<32:55,  4.74it/s]
                                                   
{'loss': 0.6537, 'grad_norm': 0.9522314667701721, 'learning_rate': 9.995451103023144e-05}

  6%|▋         | 630/10000 [04:23<32:55,  4.74it/s]
  6%|▋         | 631/10000 [04:23<33:59,  4.59it/s]
  6%|▋         | 632/10000 [04:23<33:25,  4.67it/s]
  6%|▋         | 633/10000 [04:23<32:19,  4.83it/s]
  6%|▋         | 634/10000 [04:24<31:11,  5.01it/s]
  6%|▋         | 635/10000 [04:24<30:24,  5.13it/s]
  6%|▋         | 636/10000 [04:24<29:45,  5.25it/s]
  6%|▋         | 637/10000 [04:24<29:15,  5.33it/s]
  6%|▋         | 638/10000 [04:24<30:35,  5.10it/s]
  6%|▋         | 639/10000 [04:25<33:00,  4.73it/s]
  6%|▋         | 640/10000 [04:25<33:18,  4.68it/s]
                                                   
{'loss': 0.6349, 'grad_norm': 1.0301637649536133, 'learning_rate': 9.994718641193928e-05}

  6%|▋         | 640/10000 [04:25<33:18,  4.68it/s]
  6%|▋         | 641/10000 [04:25<34:01,  4.59it/s]
  6%|▋         | 642/10000 [04:25<33:06,  4.71it/s]
  6%|▋         | 643/10000 [04:25<31:42,  4.92it/s]
  6%|▋         | 644/10000 [04:26<30:25,  5.13it/s]
  6%|▋         | 645/10000 [04:26<29:39,  5.26it/s]
  6%|▋         | 646/10000 [04:26<30:37,  5.09it/s]
  6%|▋         | 647/10000 [04:26<30:47,  5.06it/s]
  6%|▋         | 648/10000 [04:26<32:54,  4.74it/s]
  6%|▋         | 649/10000 [04:27<34:01,  4.58it/s]
  6%|▋         | 650/10000 [04:27<35:22,  4.40it/s]
                                                   
{'loss': 0.6512, 'grad_norm': 0.8331605195999146, 'learning_rate': 9.993931557921874e-05}

  6%|▋         | 650/10000 [04:27<35:22,  4.40it/s]
  7%|▋         | 651/10000 [04:27<35:28,  4.39it/s]
  7%|▋         | 652/10000 [04:27<34:46,  4.48it/s]
  7%|▋         | 653/10000 [04:28<33:34,  4.64it/s]
  7%|▋         | 654/10000 [04:28<32:35,  4.78it/s]
  7%|▋         | 655/10000 [04:28<32:45,  4.76it/s]
  7%|▋         | 656/10000 [04:28<33:37,  4.63it/s]
  7%|▋         | 657/10000 [04:28<33:24,  4.66it/s]
  7%|▋         | 658/10000 [04:29<32:52,  4.74it/s]
  7%|▋         | 659/10000 [04:29<32:08,  4.84it/s]
  7%|▋         | 660/10000 [04:29<31:26,  4.95it/s]
                                                   
{'loss': 0.6331, 'grad_norm': 0.8266019225120544, 'learning_rate': 9.993089861814402e-05}

  7%|▋         | 660/10000 [04:29<31:26,  4.95it/s]
  7%|▋         | 661/10000 [04:29<30:52,  5.04it/s]
  7%|▋         | 662/10000 [04:29<30:51,  5.04it/s]
  7%|▋         | 663/10000 [04:30<31:03,  5.01it/s]
  7%|▋         | 664/10000 [04:30<31:13,  4.98it/s]
  7%|▋         | 665/10000 [04:30<31:27,  4.95it/s]
  7%|▋         | 666/10000 [04:30<31:30,  4.94it/s]
  7%|▋         | 667/10000 [04:30<31:35,  4.92it/s]
  7%|▋         | 668/10000 [04:31<31:02,  5.01it/s]
  7%|▋         | 669/10000 [04:31<30:09,  5.16it/s]
  7%|▋         | 670/10000 [04:31<30:12,  5.15it/s]
                                                   
{'loss': 0.6369, 'grad_norm': 1.0271166563034058, 'learning_rate': 9.992193562076166e-05}

  7%|▋         | 670/10000 [04:31<30:12,  5.15it/s]
  7%|▋         | 671/10000 [04:31<31:35,  4.92it/s]
  7%|▋         | 672/10000 [04:31<32:22,  4.80it/s]
  7%|▋         | 673/10000 [04:32<33:21,  4.66it/s]
  7%|▋         | 674/10000 [04:32<33:49,  4.60it/s]
  7%|▋         | 675/10000 [04:32<34:20,  4.53it/s]
  7%|▋         | 676/10000 [04:32<33:22,  4.66it/s]
  7%|▋         | 677/10000 [04:32<32:12,  4.82it/s]
  7%|▋         | 678/10000 [04:33<31:08,  4.99it/s]
  7%|▋         | 679/10000 [04:33<33:06,  4.69it/s]
  7%|▋         | 680/10000 [04:33<34:19,  4.53it/s]
                                                   
{'loss': 0.6464, 'grad_norm': 0.7763676643371582, 'learning_rate': 9.991242668508954e-05}

  7%|▋         | 680/10000 [04:33<34:19,  4.53it/s]
  7%|▋         | 681/10000 [04:33<34:27,  4.51it/s]
  7%|▋         | 682/10000 [04:34<34:52,  4.45it/s]
  7%|▋         | 683/10000 [04:34<35:00,  4.44it/s]
  7%|▋         | 684/10000 [04:34<35:16,  4.40it/s]
  7%|▋         | 685/10000 [04:34<33:46,  4.60it/s]
  7%|▋         | 686/10000 [04:34<32:50,  4.73it/s]
  7%|▋         | 687/10000 [04:35<32:25,  4.79it/s]
  7%|▋         | 688/10000 [04:35<37:43,  4.11it/s]
  7%|▋         | 689/10000 [04:35<36:01,  4.31it/s]
  7%|▋         | 690/10000 [04:35<35:39,  4.35it/s]
                                                   
{'loss': 0.6167, 'grad_norm': 0.9032822251319885, 'learning_rate': 9.990237191511587e-05}

  7%|▋         | 690/10000 [04:35<35:39,  4.35it/s]
  7%|▋         | 691/10000 [04:36<35:33,  4.36it/s]
  7%|▋         | 692/10000 [04:36<35:30,  4.37it/s]
  7%|▋         | 693/10000 [04:36<34:22,  4.51it/s]
  7%|▋         | 694/10000 [04:36<33:09,  4.68it/s]
  7%|▋         | 695/10000 [04:36<33:23,  4.64it/s]
  7%|▋         | 696/10000 [04:37<32:35,  4.76it/s]
  7%|▋         | 697/10000 [04:37<32:25,  4.78it/s]
  7%|▋         | 698/10000 [04:37<32:55,  4.71it/s]
  7%|▋         | 699/10000 [04:37<35:10,  4.41it/s]
  7%|▋         | 700/10000 [04:38<38:21,  4.04it/s]
                                                   
{'loss': 0.6234, 'grad_norm': 0.7536276578903198, 'learning_rate': 9.989177142079802e-05}

  7%|▋         | 700/10000 [04:38<38:21,  4.04it/s]
  7%|▋         | 701/10000 [04:38<39:37,  3.91it/s]Rank 0, Worker 5: Wait for shard 45 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

  7%|▋         | 702/10000 [04:38<37:51,  4.09it/s]
  7%|▋         | 703/10000 [04:38<36:22,  4.26it/s]
  7%|▋         | 704/10000 [04:39<35:03,  4.42it/s]
  7%|▋         | 705/10000 [04:39<34:05,  4.54it/s]
  7%|▋         | 706/10000 [04:39<33:23,  4.64it/s]Rank 0, Worker 4: Wait for shard 25 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

  7%|▋         | 707/10000 [04:39<34:02,  4.55it/s]
  7%|▋         | 708/10000 [04:39<35:04,  4.42it/s]
  7%|▋         | 709/10000 [04:40<36:19,  4.26it/s]
  7%|▋         | 710/10000 [04:40<37:50,  4.09it/s]
                                                   
{'loss': 0.6182, 'grad_norm': 0.7976689338684082, 'learning_rate': 9.988062531806126e-05}

  7%|▋         | 710/10000 [04:40<37:50,  4.09it/s]
  7%|▋         | 711/10000 [04:40<38:16,  4.04it/s]
  7%|▋         | 712/10000 [04:40<37:22,  4.14it/s]
  7%|▋         | 713/10000 [04:41<36:41,  4.22it/s]
  7%|▋         | 714/10000 [04:41<36:30,  4.24it/s]
  7%|▋         | 715/10000 [04:41<36:06,  4.29it/s]
  7%|▋         | 716/10000 [04:41<35:44,  4.33it/s]
  7%|▋         | 717/10000 [04:42<35:17,  4.38it/s]Rank 0, Worker 3: Wait for shard 5 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

  7%|▋         | 718/10000 [04:42<37:34,  4.12it/s]
  7%|▋         | 719/10000 [04:42<40:30,  3.82it/s]
  7%|▋         | 720/10000 [04:42<41:44,  3.71it/s]
                                                   
{'loss': 0.6194, 'grad_norm': 0.8874609470367432, 'learning_rate': 9.986893372879762e-05}

  7%|▋         | 720/10000 [04:42<41:44,  3.71it/s]
  7%|▋         | 721/10000 [04:43<40:52,  3.78it/s]
  7%|▋         | 722/10000 [04:43<38:11,  4.05it/s]
  7%|▋         | 723/10000 [04:43<36:24,  4.25it/s]
  7%|▋         | 724/10000 [04:43<35:46,  4.32it/s]
  7%|▋         | 725/10000 [04:44<34:38,  4.46it/s]
  7%|▋         | 726/10000 [04:44<35:38,  4.34it/s]
  7%|▋         | 727/10000 [04:44<36:20,  4.25it/s]
  7%|▋         | 728/10000 [04:44<37:47,  4.09it/s]
  7%|▋         | 729/10000 [04:45<39:15,  3.94it/s]
  7%|▋         | 730/10000 [04:45<40:37,  3.80it/s]
                                                   
{'loss': 0.622, 'grad_norm': 0.8329346179962158, 'learning_rate': 9.985669678086443e-05}

  7%|▋         | 730/10000 [04:45<40:37,  3.80it/s]
  7%|▋         | 731/10000 [04:45<41:36,  3.71it/s]
  7%|▋         | 732/10000 [04:45<39:27,  3.91it/s]
  7%|▋         | 733/10000 [04:46<38:37,  4.00it/s]
  7%|▋         | 734/10000 [04:46<38:43,  3.99it/s]
  7%|▋         | 735/10000 [04:46<38:54,  3.97it/s]
  7%|▋         | 736/10000 [04:46<37:24,  4.13it/s]
  7%|▋         | 737/10000 [04:47<37:24,  4.13it/s]
  7%|▋         | 738/10000 [04:47<36:29,  4.23it/s]
  7%|▋         | 739/10000 [04:47<38:15,  4.03it/s]
  7%|▋         | 740/10000 [04:47<37:21,  4.13it/s]
                                                   
{'loss': 0.6179, 'grad_norm': 1.1681642532348633, 'learning_rate': 9.984391460808298e-05}

  7%|▋         | 740/10000 [04:47<37:21,  4.13it/s]
  7%|▋         | 741/10000 [04:48<37:31,  4.11it/s]
  7%|▋         | 742/10000 [04:48<37:05,  4.16it/s]
  7%|▋         | 743/10000 [04:48<36:42,  4.20it/s]
  7%|▋         | 744/10000 [04:48<36:30,  4.22it/s]
  7%|▋         | 745/10000 [04:48<35:42,  4.32it/s]
  7%|▋         | 746/10000 [04:49<34:56,  4.41it/s]
  7%|▋         | 747/10000 [04:49<34:42,  4.44it/s]
  7%|▋         | 748/10000 [04:49<34:54,  4.42it/s]
  7%|▋         | 749/10000 [04:49<35:06,  4.39it/s]
  8%|▊         | 750/10000 [04:50<34:33,  4.46it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.14 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 18.28 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.46 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 18.57 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 18.80 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.00 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 750: eval/loss = 0.6140 (38s)

                                                   
{'loss': 0.6225, 'grad_norm': 0.7663777470588684, 'learning_rate': 9.983058735023709e-05}

  8%|▊         | 750/10000 [05:27<34:33,  4.46it/s]
  8%|▊         | 751/10000 [05:27<29:27:03, 11.46s/it]
  8%|▊         | 752/10000 [05:27<20:45:30,  8.08s/it]
  8%|▊         | 753/10000 [05:28<14:40:25,  5.71s/it]
  8%|▊         | 754/10000 [05:28<10:25:01,  4.06s/it]
  8%|▊         | 755/10000 [05:28<7:26:27,  2.90s/it] 
  8%|▊         | 756/10000 [05:28<5:22:40,  2.09s/it]
  8%|▊         | 757/10000 [05:28<3:55:37,  1.53s/it]
  8%|▊         | 758/10000 [05:29<2:54:45,  1.13s/it]
  8%|▊         | 759/10000 [05:29<2:11:55,  1.17it/s]
  8%|▊         | 760/10000 [05:29<1:41:41,  1.51it/s]
                                                     
{'loss': 0.6143, 'grad_norm': 1.088477611541748, 'learning_rate': 9.98167151530715e-05}

  8%|▊         | 760/10000 [05:29<1:41:41,  1.51it/s]
  8%|▊         | 761/10000 [05:29<1:21:09,  1.90it/s]
  8%|▊         | 762/10000 [05:29<1:06:28,  2.32it/s]
  8%|▊         | 763/10000 [05:30<55:45,  2.76it/s]  
  8%|▊         | 764/10000 [05:30<49:02,  3.14it/s]
  8%|▊         | 765/10000 [05:30<44:25,  3.46it/s]
  8%|▊         | 766/10000 [05:30<41:12,  3.74it/s]
  8%|▊         | 767/10000 [05:31<38:59,  3.95it/s]
  8%|▊         | 768/10000 [05:31<37:26,  4.11it/s]
  8%|▊         | 769/10000 [05:31<35:50,  4.29it/s]
  8%|▊         | 770/10000 [05:31<34:38,  4.44it/s]
                                                   
{'loss': 0.5978, 'grad_norm': 0.681723415851593, 'learning_rate': 9.980229816829034e-05}

  8%|▊         | 770/10000 [05:31<34:38,  4.44it/s]
  8%|▊         | 771/10000 [05:31<34:52,  4.41it/s]
  8%|▊         | 772/10000 [05:32<33:55,  4.53it/s]
  8%|▊         | 773/10000 [05:32<33:32,  4.59it/s]
  8%|▊         | 774/10000 [05:32<33:21,  4.61it/s]
  8%|▊         | 775/10000 [05:32<33:34,  4.58it/s]
  8%|▊         | 776/10000 [05:32<33:27,  4.60it/s]
  8%|▊         | 777/10000 [05:33<32:43,  4.70it/s]
  8%|▊         | 778/10000 [05:33<33:42,  4.56it/s]
  8%|▊         | 779/10000 [05:33<33:08,  4.64it/s]
  8%|▊         | 780/10000 [05:33<33:05,  4.64it/s]
                                                   
{'loss': 0.61, 'grad_norm': 0.8217130303382874, 'learning_rate': 9.978733655355544e-05}

  8%|▊         | 780/10000 [05:33<33:05,  4.64it/s]
  8%|▊         | 781/10000 [05:34<33:42,  4.56it/s]
  8%|▊         | 782/10000 [05:34<34:01,  4.52it/s]
  8%|▊         | 783/10000 [05:34<34:38,  4.43it/s]
  8%|▊         | 784/10000 [05:34<34:53,  4.40it/s]
  8%|▊         | 785/10000 [05:34<33:42,  4.56it/s]
  8%|▊         | 786/10000 [05:35<33:02,  4.65it/s]
  8%|▊         | 787/10000 [05:35<32:43,  4.69it/s]
  8%|▊         | 788/10000 [05:35<32:15,  4.76it/s]
  8%|▊         | 789/10000 [05:35<33:24,  4.59it/s]
  8%|▊         | 790/10000 [05:36<34:57,  4.39it/s]
                                                   
{'loss': 0.6067, 'grad_norm': 0.7286795377731323, 'learning_rate': 9.977183047248464e-05}

  8%|▊         | 790/10000 [05:36<34:57,  4.39it/s]
  8%|▊         | 791/10000 [05:36<36:39,  4.19it/s]
  8%|▊         | 792/10000 [05:36<37:07,  4.13it/s]
  8%|▊         | 793/10000 [05:36<35:40,  4.30it/s]
  8%|▊         | 794/10000 [05:37<34:11,  4.49it/s]
  8%|▊         | 795/10000 [05:37<33:30,  4.58it/s]
  8%|▊         | 796/10000 [05:37<33:17,  4.61it/s]
  8%|▊         | 797/10000 [05:37<32:01,  4.79it/s]
  8%|▊         | 798/10000 [05:37<32:01,  4.79it/s]
  8%|▊         | 799/10000 [05:38<32:35,  4.70it/s]
  8%|▊         | 800/10000 [05:38<32:57,  4.65it/s]
                                                   
{'loss': 0.5884, 'grad_norm': 0.6988299489021301, 'learning_rate': 9.975578009464992e-05}

  8%|▊         | 800/10000 [05:38<32:57,  4.65it/s]
  8%|▊         | 801/10000 [05:38<33:39,  4.56it/s]
  8%|▊         | 802/10000 [05:38<33:11,  4.62it/s]
  8%|▊         | 803/10000 [05:38<32:44,  4.68it/s]
  8%|▊         | 804/10000 [05:39<32:22,  4.73it/s]
  8%|▊         | 805/10000 [05:39<31:38,  4.84it/s]Rank 0, Worker 1: Wait for shard 65 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

  8%|▊         | 806/10000 [05:39<31:33,  4.85it/s]
  8%|▊         | 807/10000 [05:39<31:31,  4.86it/s]
  8%|▊         | 808/10000 [05:39<31:52,  4.81it/s]
  8%|▊         | 809/10000 [05:40<32:08,  4.77it/s]
  8%|▊         | 810/10000 [05:40<32:09,  4.76it/s]
                                                   
{'loss': 0.5876, 'grad_norm': 1.0712876319885254, 'learning_rate': 9.97391855955757e-05}

  8%|▊         | 810/10000 [05:40<32:09,  4.76it/s]Rank 0, Worker 0: Wait for shard 11 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

  8%|▊         | 811/10000 [05:40<33:20,  4.59it/s]
  8%|▊         | 812/10000 [05:40<33:40,  4.55it/s]Rank 0, Worker 2: Wait for shard 34 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

  8%|▊         | 813/10000 [05:41<35:57,  4.26it/s]
  8%|▊         | 814/10000 [05:41<38:41,  3.96it/s]
  8%|▊         | 815/10000 [05:41<38:47,  3.95it/s]
  8%|▊         | 816/10000 [05:41<39:01,  3.92it/s]
  8%|▊         | 817/10000 [05:42<39:29,  3.87it/s]
  8%|▊         | 818/10000 [05:42<39:26,  3.88it/s]
  8%|▊         | 819/10000 [05:42<39:41,  3.86it/s]
  8%|▊         | 820/10000 [05:42<39:27,  3.88it/s]
                                                   
{'loss': 0.6029, 'grad_norm': 0.7668588161468506, 'learning_rate': 9.972204715673669e-05}

  8%|▊         | 820/10000 [05:42<39:27,  3.88it/s]
  8%|▊         | 821/10000 [05:43<38:29,  3.98it/s]
  8%|▊         | 822/10000 [05:43<37:50,  4.04it/s]
  8%|▊         | 823/10000 [05:43<37:54,  4.03it/s]
  8%|▊         | 824/10000 [05:43<37:43,  4.05it/s]
  8%|▊         | 825/10000 [05:44<37:05,  4.12it/s]
  8%|▊         | 826/10000 [05:44<35:10,  4.35it/s]
  8%|▊         | 827/10000 [05:44<35:48,  4.27it/s]
  8%|▊         | 828/10000 [05:44<35:51,  4.26it/s]
  8%|▊         | 829/10000 [05:45<34:58,  4.37it/s]
  8%|▊         | 830/10000 [05:45<34:14,  4.46it/s]
                                                   
{'loss': 0.5796, 'grad_norm': 0.9838575720787048, 'learning_rate': 9.970436496555617e-05}

  8%|▊         | 830/10000 [05:45<34:14,  4.46it/s]
  8%|▊         | 831/10000 [05:45<35:21,  4.32it/s]
  8%|▊         | 832/10000 [05:45<35:34,  4.30it/s]
  8%|▊         | 833/10000 [05:45<35:49,  4.26it/s]
  8%|▊         | 834/10000 [05:46<35:34,  4.29it/s]
  8%|▊         | 835/10000 [05:46<34:47,  4.39it/s]
  8%|▊         | 836/10000 [05:46<35:11,  4.34it/s]
  8%|▊         | 837/10000 [05:46<35:41,  4.28it/s]
  8%|▊         | 838/10000 [05:47<36:13,  4.22it/s]
  8%|▊         | 839/10000 [05:47<36:36,  4.17it/s]
  8%|▊         | 840/10000 [05:47<37:17,  4.09it/s]
                                                   
{'loss': 0.5831, 'grad_norm': 0.8770014047622681, 'learning_rate': 9.968613921540373e-05}

  8%|▊         | 840/10000 [05:47<37:17,  4.09it/s]
  8%|▊         | 841/10000 [05:47<37:06,  4.11it/s]
  8%|▊         | 842/10000 [05:48<37:13,  4.10it/s]
  8%|▊         | 843/10000 [05:48<36:52,  4.14it/s]
  8%|▊         | 844/10000 [05:48<37:02,  4.12it/s]
  8%|▊         | 845/10000 [05:48<36:44,  4.15it/s]
  8%|▊         | 846/10000 [05:49<39:08,  3.90it/s]
  8%|▊         | 847/10000 [05:49<38:02,  4.01it/s]
  8%|▊         | 848/10000 [05:49<39:25,  3.87it/s]
  8%|▊         | 849/10000 [05:49<38:28,  3.96it/s]
  8%|▊         | 850/10000 [05:50<38:04,  4.01it/s]
                                                   
{'loss': 0.5828, 'grad_norm': 0.935934841632843, 'learning_rate': 9.966737010559326e-05}

  8%|▊         | 850/10000 [05:50<38:04,  4.01it/s]
  9%|▊         | 851/10000 [05:50<39:25,  3.87it/s]
  9%|▊         | 852/10000 [05:50<38:30,  3.96it/s]
  9%|▊         | 853/10000 [05:50<36:47,  4.14it/s]
  9%|▊         | 854/10000 [05:51<35:13,  4.33it/s]
  9%|▊         | 855/10000 [05:51<34:07,  4.47it/s]
  9%|▊         | 856/10000 [05:51<33:07,  4.60it/s]
  9%|▊         | 857/10000 [05:51<31:58,  4.77it/s]
  9%|▊         | 858/10000 [05:51<31:12,  4.88it/s]
  9%|▊         | 859/10000 [05:52<31:22,  4.85it/s]
  9%|▊         | 860/10000 [05:52<31:40,  4.81it/s]
                                                   
{'loss': 0.5872, 'grad_norm': 0.8480046391487122, 'learning_rate': 9.964805784138072e-05}

  9%|▊         | 860/10000 [05:52<31:40,  4.81it/s]
  9%|▊         | 861/10000 [05:52<32:48,  4.64it/s]
  9%|▊         | 862/10000 [05:52<32:24,  4.70it/s]
  9%|▊         | 863/10000 [05:52<31:53,  4.78it/s]
  9%|▊         | 864/10000 [05:53<31:35,  4.82it/s]
  9%|▊         | 865/10000 [05:53<31:11,  4.88it/s]
  9%|▊         | 866/10000 [05:53<30:53,  4.93it/s]
  9%|▊         | 867/10000 [05:53<30:49,  4.94it/s]
  9%|▊         | 868/10000 [05:53<32:20,  4.71it/s]
  9%|▊         | 869/10000 [05:54<32:04,  4.74it/s]
  9%|▊         | 870/10000 [05:54<31:45,  4.79it/s]
                                                   
{'loss': 0.5963, 'grad_norm': 0.7544788122177124, 'learning_rate': 9.962820263396195e-05}

  9%|▊         | 870/10000 [05:54<31:45,  4.79it/s]
  9%|▊         | 871/10000 [05:54<32:29,  4.68it/s]
  9%|▊         | 872/10000 [05:54<34:30,  4.41it/s]
  9%|▊         | 873/10000 [05:55<36:08,  4.21it/s]
  9%|▊         | 874/10000 [05:55<34:49,  4.37it/s]
  9%|▉         | 875/10000 [05:55<33:39,  4.52it/s]
  9%|▉         | 876/10000 [05:55<33:19,  4.56it/s]
  9%|▉         | 877/10000 [05:55<32:52,  4.62it/s]
  9%|▉         | 878/10000 [05:56<33:52,  4.49it/s]
  9%|▉         | 879/10000 [05:56<34:12,  4.44it/s]
  9%|▉         | 880/10000 [05:56<33:39,  4.52it/s]
                                                   
{'loss': 0.5789, 'grad_norm': 0.9721704125404358, 'learning_rate': 9.960780470047033e-05}

  9%|▉         | 880/10000 [05:56<33:39,  4.52it/s]
  9%|▉         | 881/10000 [05:56<33:15,  4.57it/s]
  9%|▉         | 882/10000 [05:57<32:31,  4.67it/s]
  9%|▉         | 883/10000 [05:57<32:09,  4.72it/s]
  9%|▉         | 884/10000 [05:57<32:22,  4.69it/s]
  9%|▉         | 885/10000 [05:57<32:27,  4.68it/s]
  9%|▉         | 886/10000 [05:57<31:51,  4.77it/s]
  9%|▉         | 887/10000 [05:58<31:24,  4.84it/s]
  9%|▉         | 888/10000 [05:58<30:56,  4.91it/s]
  9%|▉         | 889/10000 [05:58<30:40,  4.95it/s]
  9%|▉         | 890/10000 [05:58<30:43,  4.94it/s]
                                                   
{'loss': 0.5674, 'grad_norm': 0.7874745726585388, 'learning_rate': 9.958686426397437e-05}

  9%|▉         | 890/10000 [05:58<30:43,  4.94it/s]
  9%|▉         | 891/10000 [05:58<30:57,  4.90it/s]
  9%|▉         | 892/10000 [05:59<31:12,  4.86it/s]
  9%|▉         | 893/10000 [05:59<32:25,  4.68it/s]
  9%|▉         | 894/10000 [05:59<32:17,  4.70it/s]
  9%|▉         | 895/10000 [05:59<32:50,  4.62it/s]
  9%|▉         | 896/10000 [05:59<32:21,  4.69it/s]
  9%|▉         | 897/10000 [06:00<32:36,  4.65it/s]
  9%|▉         | 898/10000 [06:00<33:20,  4.55it/s]
  9%|▉         | 899/10000 [06:00<32:33,  4.66it/s]
  9%|▉         | 900/10000 [06:00<32:13,  4.71it/s]
                                                   
{'loss': 0.5837, 'grad_norm': 0.9333078861236572, 'learning_rate': 9.956538155347534e-05}

  9%|▉         | 900/10000 [06:00<32:13,  4.71it/s]
  9%|▉         | 901/10000 [06:01<32:13,  4.71it/s]
  9%|▉         | 902/10000 [06:01<31:41,  4.78it/s]
  9%|▉         | 903/10000 [06:01<31:14,  4.85it/s]
  9%|▉         | 904/10000 [06:01<30:53,  4.91it/s]
  9%|▉         | 905/10000 [06:01<30:52,  4.91it/s]
  9%|▉         | 906/10000 [06:02<31:00,  4.89it/s]
  9%|▉         | 907/10000 [06:02<32:03,  4.73it/s]
  9%|▉         | 908/10000 [06:02<32:16,  4.70it/s]
  9%|▉         | 909/10000 [06:02<31:47,  4.77it/s]
  9%|▉         | 910/10000 [06:02<31:34,  4.80it/s]
                                                   
{'loss': 0.5892, 'grad_norm': 0.8755942583084106, 'learning_rate': 9.95433568039047e-05}

  9%|▉         | 910/10000 [06:02<31:34,  4.80it/s]
  9%|▉         | 911/10000 [06:03<31:38,  4.79it/s]
  9%|▉         | 912/10000 [06:03<31:25,  4.82it/s]
  9%|▉         | 913/10000 [06:03<31:17,  4.84it/s]
  9%|▉         | 914/10000 [06:03<30:55,  4.90it/s]
  9%|▉         | 915/10000 [06:03<30:36,  4.95it/s]
  9%|▉         | 916/10000 [06:04<30:32,  4.96it/s]
  9%|▉         | 917/10000 [06:04<30:35,  4.95it/s]
  9%|▉         | 918/10000 [06:04<31:23,  4.82it/s]
  9%|▉         | 919/10000 [06:04<31:31,  4.80it/s]
  9%|▉         | 920/10000 [06:04<32:17,  4.69it/s]
                                                   
{'loss': 0.5908, 'grad_norm': 0.914348304271698, 'learning_rate': 9.952079025612162e-05}

  9%|▉         | 920/10000 [06:05<32:17,  4.69it/s]
  9%|▉         | 921/10000 [06:05<31:52,  4.75it/s]
  9%|▉         | 922/10000 [06:05<32:33,  4.65it/s]
  9%|▉         | 923/10000 [06:05<32:53,  4.60it/s]
  9%|▉         | 924/10000 [06:05<32:11,  4.70it/s]
  9%|▉         | 925/10000 [06:06<31:57,  4.73it/s]
  9%|▉         | 926/10000 [06:06<31:26,  4.81it/s]
  9%|▉         | 927/10000 [06:06<31:17,  4.83it/s]
  9%|▉         | 928/10000 [06:06<31:17,  4.83it/s]
  9%|▉         | 929/10000 [06:06<31:04,  4.86it/s]
  9%|▉         | 930/10000 [06:07<31:13,  4.84it/s]
                                                   
{'loss': 0.5838, 'grad_norm': 0.7876885533332825, 'learning_rate': 9.949768215691022e-05}

  9%|▉         | 930/10000 [06:07<31:13,  4.84it/s]
  9%|▉         | 931/10000 [06:07<31:27,  4.80it/s]
  9%|▉         | 932/10000 [06:07<31:00,  4.87it/s]
  9%|▉         | 933/10000 [06:07<30:47,  4.91it/s]
  9%|▉         | 934/10000 [06:07<30:49,  4.90it/s]
  9%|▉         | 935/10000 [06:08<30:45,  4.91it/s]
  9%|▉         | 936/10000 [06:08<30:45,  4.91it/s]
  9%|▉         | 937/10000 [06:08<30:31,  4.95it/s]
  9%|▉         | 938/10000 [06:08<30:27,  4.96it/s]
  9%|▉         | 939/10000 [06:08<30:25,  4.96it/s]
  9%|▉         | 940/10000 [06:09<31:24,  4.81it/s]
                                                   
{'loss': 0.5608, 'grad_norm': 0.7325435280799866, 'learning_rate': 9.9474032758977e-05}

  9%|▉         | 940/10000 [06:09<31:24,  4.81it/s]
  9%|▉         | 941/10000 [06:09<32:02,  4.71it/s]
  9%|▉         | 942/10000 [06:09<32:08,  4.70it/s]
  9%|▉         | 943/10000 [06:09<32:03,  4.71it/s]
  9%|▉         | 944/10000 [06:09<32:09,  4.69it/s]
  9%|▉         | 945/10000 [06:10<31:38,  4.77it/s]
  9%|▉         | 946/10000 [06:10<31:10,  4.84it/s]
  9%|▉         | 947/10000 [06:10<31:05,  4.85it/s]
  9%|▉         | 948/10000 [06:10<31:08,  4.84it/s]
  9%|▉         | 949/10000 [06:10<30:46,  4.90it/s]
 10%|▉         | 950/10000 [06:11<30:30,  4.94it/s]
                                                   
{'loss': 0.544, 'grad_norm': 0.6841356754302979, 'learning_rate': 9.944984232094794e-05}

 10%|▉         | 950/10000 [06:11<30:30,  4.94it/s]
 10%|▉         | 951/10000 [06:11<30:54,  4.88it/s]
 10%|▉         | 952/10000 [06:11<31:07,  4.85it/s]
 10%|▉         | 953/10000 [06:11<31:02,  4.86it/s]
 10%|▉         | 954/10000 [06:12<31:45,  4.75it/s]
 10%|▉         | 955/10000 [06:12<31:44,  4.75it/s]
 10%|▉         | 956/10000 [06:12<31:32,  4.78it/s]
 10%|▉         | 957/10000 [06:12<31:54,  4.72it/s]
 10%|▉         | 958/10000 [06:12<31:26,  4.79it/s]
 10%|▉         | 959/10000 [06:13<33:37,  4.48it/s]
 10%|▉         | 960/10000 [06:13<32:32,  4.63it/s]
                                                   
{'loss': 0.5795, 'grad_norm': 1.0104835033416748, 'learning_rate': 9.942511110736584e-05}

 10%|▉         | 960/10000 [06:13<32:32,  4.63it/s]
 10%|▉         | 961/10000 [06:13<32:02,  4.70it/s]
 10%|▉         | 962/10000 [06:13<31:36,  4.77it/s]
 10%|▉         | 963/10000 [06:13<31:06,  4.84it/s]
 10%|▉         | 964/10000 [06:14<30:50,  4.88it/s]
 10%|▉         | 965/10000 [06:14<30:53,  4.87it/s]
 10%|▉         | 966/10000 [06:14<30:58,  4.86it/s]
 10%|▉         | 967/10000 [06:14<30:39,  4.91it/s]
 10%|▉         | 968/10000 [06:14<30:13,  4.98it/s]
 10%|▉         | 969/10000 [06:15<30:05,  5.00it/s]
 10%|▉         | 970/10000 [06:15<30:03,  5.01it/s]
                                                   
{'loss': 0.5541, 'grad_norm': 0.756842851638794, 'learning_rate': 9.939983938868726e-05}

 10%|▉         | 970/10000 [06:15<30:03,  5.01it/s]
 10%|▉         | 971/10000 [06:15<30:15,  4.97it/s]
 10%|▉         | 972/10000 [06:15<30:06,  5.00it/s]
 10%|▉         | 973/10000 [06:15<29:58,  5.02it/s]
 10%|▉         | 974/10000 [06:16<30:11,  4.98it/s]
 10%|▉         | 975/10000 [06:16<30:58,  4.86it/s]
 10%|▉         | 976/10000 [06:16<30:48,  4.88it/s]
 10%|▉         | 977/10000 [06:16<30:41,  4.90it/s]
 10%|▉         | 978/10000 [06:16<30:36,  4.91it/s]
 10%|▉         | 979/10000 [06:17<30:19,  4.96it/s]
 10%|▉         | 980/10000 [06:17<30:03,  5.00it/s]
                                                   
{'loss': 0.5495, 'grad_norm': 0.8415982127189636, 'learning_rate': 9.93740274412797e-05}

 10%|▉         | 980/10000 [06:17<30:03,  5.00it/s]
 10%|▉         | 981/10000 [06:17<30:25,  4.94it/s]
 10%|▉         | 982/10000 [06:17<30:14,  4.97it/s]
 10%|▉         | 983/10000 [06:17<30:16,  4.96it/s]
 10%|▉         | 984/10000 [06:18<30:26,  4.94it/s]
 10%|▉         | 985/10000 [06:18<30:27,  4.93it/s]
 10%|▉         | 986/10000 [06:18<30:23,  4.94it/s]
 10%|▉         | 987/10000 [06:18<30:22,  4.95it/s]
 10%|▉         | 988/10000 [06:18<30:16,  4.96it/s]
 10%|▉         | 989/10000 [06:19<30:11,  4.97it/s]
 10%|▉         | 990/10000 [06:19<30:17,  4.96it/s]
                                                   
{'loss': 0.569, 'grad_norm': 0.9179241061210632, 'learning_rate': 9.934767554741846e-05}

 10%|▉         | 990/10000 [06:19<30:17,  4.96it/s]
 10%|▉         | 991/10000 [06:19<31:17,  4.80it/s]
 10%|▉         | 992/10000 [06:19<31:43,  4.73it/s]
 10%|▉         | 993/10000 [06:20<32:13,  4.66it/s]
 10%|▉         | 994/10000 [06:20<30:53,  4.86it/s]
 10%|▉         | 995/10000 [06:20<30:34,  4.91it/s]
 10%|▉         | 996/10000 [06:20<30:14,  4.96it/s]
 10%|▉         | 997/10000 [06:20<30:46,  4.88it/s]
 10%|▉         | 998/10000 [06:21<30:58,  4.84it/s]
 10%|▉         | 999/10000 [06:21<32:05,  4.68it/s]
 10%|█         | 1000/10000 [06:21<31:19,  4.79it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 19.66 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 21.25 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 21.44 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 21.95 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 22.67 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 22.76 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 1000: eval/loss = 0.5366 (46s)

                                                    
{'loss': 0.5625, 'grad_norm': 0.8125057220458984, 'learning_rate': 9.932078399528361e-05}

 10%|█         | 1000/10000 [07:07<31:19,  4.79it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-1000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-1000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-1000/wandb_config.json

 10%|█         | 1001/10000 [07:37<57:05:08, 22.84s/it]
 10%|█         | 1002/10000 [07:37<40:06:31, 16.05s/it]
 10%|█         | 1003/10000 [07:37<28:14:25, 11.30s/it]
 10%|█         | 1004/10000 [07:37<19:55:26,  7.97s/it]
 10%|█         | 1005/10000 [07:37<14:05:39,  5.64s/it]
 10%|█         | 1006/10000 [07:38<10:00:32,  4.01s/it]
 10%|█         | 1007/10000 [07:38<7:09:04,  2.86s/it] 
 10%|█         | 1008/10000 [07:38<5:10:02,  2.07s/it]
 10%|█         | 1009/10000 [07:38<3:46:51,  1.51s/it]
 10%|█         | 1010/10000 [07:39<2:49:16,  1.13s/it]
                                                      
{'loss': 0.5586, 'grad_norm': 0.9515894651412964, 'learning_rate': 9.929335307895689e-05}

 10%|█         | 1010/10000 [07:39<2:49:16,  1.13s/it]
 10%|█         | 1011/10000 [07:39<2:10:13,  1.15it/s]
 10%|█         | 1012/10000 [07:39<1:40:49,  1.49it/s]
 10%|█         | 1013/10000 [07:39<1:20:13,  1.87it/s]
 10%|█         | 1014/10000 [07:39<1:05:30,  2.29it/s]
 10%|█         | 1015/10000 [07:40<54:31,  2.75it/s]  
 10%|█         | 1016/10000 [07:40<47:15,  3.17it/s]
 10%|█         | 1017/10000 [07:40<42:01,  3.56it/s]
 10%|█         | 1018/10000 [07:40<38:28,  3.89it/s]
 10%|█         | 1019/10000 [07:40<36:54,  4.06it/s]
 10%|█         | 1020/10000 [07:41<35:51,  4.17it/s]
                                                    
{'loss': 0.5423, 'grad_norm': 0.8592225909233093, 'learning_rate': 9.926538309841839e-05}

 10%|█         | 1020/10000 [07:41<35:51,  4.17it/s]
 10%|█         | 1021/10000 [07:41<36:17,  4.12it/s]
 10%|█         | 1022/10000 [07:41<35:07,  4.26it/s]
 10%|█         | 1023/10000 [07:41<34:46,  4.30it/s]
 10%|█         | 1024/10000 [07:42<33:08,  4.51it/s]
 10%|█         | 1025/10000 [07:42<32:19,  4.63it/s]
 10%|█         | 1026/10000 [07:42<31:38,  4.73it/s]
 10%|█         | 1027/10000 [07:42<31:10,  4.80it/s]
 10%|█         | 1028/10000 [07:42<31:11,  4.79it/s]
 10%|█         | 1029/10000 [07:43<31:44,  4.71it/s]
 10%|█         | 1030/10000 [07:43<31:26,  4.76it/s]
                                                    
{'loss': 0.5434, 'grad_norm': 0.9244703054428101, 'learning_rate': 9.923687435954334e-05}

 10%|█         | 1030/10000 [07:43<31:26,  4.76it/s]
 10%|█         | 1031/10000 [07:43<31:32,  4.74it/s]
 10%|█         | 1032/10000 [07:43<30:59,  4.82it/s]
 10%|█         | 1033/10000 [07:43<30:31,  4.90it/s]
 10%|█         | 1034/10000 [07:44<30:20,  4.92it/s]
 10%|█         | 1035/10000 [07:44<30:14,  4.94it/s]
 10%|█         | 1036/10000 [07:44<30:41,  4.87it/s]
 10%|█         | 1037/10000 [07:44<31:12,  4.79it/s]
 10%|█         | 1038/10000 [07:44<31:20,  4.76it/s]
 10%|█         | 1039/10000 [07:45<31:14,  4.78it/s]
 10%|█         | 1040/10000 [07:45<30:44,  4.86it/s]
                                                    
{'loss': 0.5401, 'grad_norm': 0.9668888449668884, 'learning_rate': 9.920782717409873e-05}

 10%|█         | 1040/10000 [07:45<30:44,  4.86it/s]
 10%|█         | 1041/10000 [07:45<30:35,  4.88it/s]
 10%|█         | 1042/10000 [07:45<30:24,  4.91it/s]
 10%|█         | 1043/10000 [07:45<30:16,  4.93it/s]
 10%|█         | 1044/10000 [07:46<30:20,  4.92it/s]
 10%|█         | 1045/10000 [07:46<31:06,  4.80it/s]
 10%|█         | 1046/10000 [07:46<30:57,  4.82it/s]
 10%|█         | 1047/10000 [07:46<30:35,  4.88it/s]
 10%|█         | 1048/10000 [07:46<30:17,  4.92it/s]
 10%|█         | 1049/10000 [07:47<30:03,  4.96it/s]
 10%|█         | 1050/10000 [07:47<30:00,  4.97it/s]
                                                    
{'loss': 0.5231, 'grad_norm': 0.7562168836593628, 'learning_rate': 9.917824185973994e-05}

 10%|█         | 1050/10000 [07:47<30:00,  4.97it/s]
 11%|█         | 1051/10000 [07:47<30:31,  4.89it/s]
 11%|█         | 1052/10000 [07:47<30:31,  4.89it/s]
 11%|█         | 1053/10000 [07:47<30:24,  4.90it/s]
 11%|█         | 1054/10000 [07:48<30:21,  4.91it/s]
 11%|█         | 1055/10000 [07:48<30:17,  4.92it/s]
 11%|█         | 1056/10000 [07:48<30:28,  4.89it/s]
 11%|█         | 1057/10000 [07:48<30:02,  4.96it/s]
 11%|█         | 1058/10000 [07:48<29:53,  4.99it/s]
 11%|█         | 1059/10000 [07:49<29:58,  4.97it/s]
 11%|█         | 1060/10000 [07:49<30:19,  4.91it/s]
                                                    
{'loss': 0.527, 'grad_norm': 0.6973814368247986, 'learning_rate': 9.914811874000723e-05}

 11%|█         | 1060/10000 [07:49<30:19,  4.91it/s]
 11%|█         | 1061/10000 [07:49<31:54,  4.67it/s]
 11%|█         | 1062/10000 [07:49<32:43,  4.55it/s]
 11%|█         | 1063/10000 [07:50<32:37,  4.57it/s]
 11%|█         | 1064/10000 [07:50<31:41,  4.70it/s]
 11%|█         | 1065/10000 [07:50<31:11,  4.77it/s]
 11%|█         | 1066/10000 [07:50<30:26,  4.89it/s]
 11%|█         | 1067/10000 [07:50<30:10,  4.93it/s]
 11%|█         | 1068/10000 [07:51<29:41,  5.02it/s]
 11%|█         | 1069/10000 [07:51<29:28,  5.05it/s]
 11%|█         | 1070/10000 [07:51<29:11,  5.10it/s]
                                                    
{'loss': 0.5114, 'grad_norm': 0.959357500076294, 'learning_rate': 9.911745814432218e-05}

 11%|█         | 1070/10000 [07:51<29:11,  5.10it/s]
 11%|█         | 1071/10000 [07:51<29:20,  5.07it/s]
 11%|█         | 1072/10000 [07:51<28:39,  5.19it/s]
 11%|█         | 1073/10000 [07:52<28:13,  5.27it/s]
 11%|█         | 1074/10000 [07:52<28:00,  5.31it/s]
 11%|█         | 1075/10000 [07:52<27:57,  5.32it/s]
 11%|█         | 1076/10000 [07:52<28:04,  5.30it/s]
 11%|█         | 1077/10000 [07:52<28:16,  5.26it/s]
 11%|█         | 1078/10000 [07:52<28:10,  5.28it/s]
 11%|█         | 1079/10000 [07:53<28:31,  5.21it/s]
 11%|█         | 1080/10000 [07:53<29:25,  5.05it/s]
                                                    
{'loss': 0.5138, 'grad_norm': 0.8211817741394043, 'learning_rate': 9.90862604079842e-05}

 11%|█         | 1080/10000 [07:53<29:25,  5.05it/s]
 11%|█         | 1081/10000 [07:53<29:25,  5.05it/s]
 11%|█         | 1082/10000 [07:53<28:37,  5.19it/s]
 11%|█         | 1083/10000 [07:53<28:18,  5.25it/s]
 11%|█         | 1084/10000 [07:54<28:35,  5.20it/s]
 11%|█         | 1085/10000 [07:54<28:50,  5.15it/s]
 11%|█         | 1086/10000 [07:54<29:01,  5.12it/s]
 11%|█         | 1087/10000 [07:54<29:15,  5.08it/s]
 11%|█         | 1088/10000 [07:54<29:20,  5.06it/s]
 11%|█         | 1089/10000 [07:55<29:23,  5.05it/s]
 11%|█         | 1090/10000 [07:55<29:33,  5.02it/s]
                                                    
{'loss': 0.5104, 'grad_norm': 0.8965429663658142, 'learning_rate': 9.90545258721667e-05}

 11%|█         | 1090/10000 [07:55<29:33,  5.02it/s]
 11%|█         | 1091/10000 [07:55<30:30,  4.87it/s]
 11%|█         | 1092/10000 [07:55<30:44,  4.83it/s]
 11%|█         | 1093/10000 [07:55<30:56,  4.80it/s]
 11%|█         | 1094/10000 [07:56<31:14,  4.75it/s]
 11%|█         | 1095/10000 [07:56<31:29,  4.71it/s]
 11%|█         | 1096/10000 [07:56<32:17,  4.59it/s]
 11%|█         | 1097/10000 [07:56<32:29,  4.57it/s]
 11%|█         | 1098/10000 [07:57<33:18,  4.45it/s]
 11%|█         | 1099/10000 [07:57<33:08,  4.48it/s]
 11%|█         | 1100/10000 [07:57<32:52,  4.51it/s]
                                                    
{'loss': 0.5073, 'grad_norm': 0.9305912852287292, 'learning_rate': 9.90222548839135e-05}

 11%|█         | 1100/10000 [07:57<32:52,  4.51it/s]
 11%|█         | 1101/10000 [07:57<32:49,  4.52it/s]
 11%|█         | 1102/10000 [07:57<32:23,  4.58it/s]
 11%|█         | 1103/10000 [07:58<32:21,  4.58it/s]
 11%|█         | 1104/10000 [07:58<31:31,  4.70it/s]
 11%|█         | 1105/10000 [07:58<30:50,  4.81it/s]
 11%|█         | 1106/10000 [07:58<30:23,  4.88it/s]
 11%|█         | 1107/10000 [07:58<30:14,  4.90it/s]
 11%|█         | 1108/10000 [07:59<30:15,  4.90it/s]
 11%|█         | 1109/10000 [07:59<31:31,  4.70it/s]
 11%|█         | 1110/10000 [07:59<32:08,  4.61it/s]
                                                    
{'loss': 0.5068, 'grad_norm': 0.7826094627380371, 'learning_rate': 9.898944779613495e-05}

 11%|█         | 1110/10000 [07:59<32:08,  4.61it/s]
 11%|█         | 1111/10000 [07:59<33:15,  4.46it/s]
 11%|█         | 1112/10000 [08:00<35:06,  4.22it/s]
 11%|█         | 1113/10000 [08:00<34:59,  4.23it/s]
 11%|█         | 1114/10000 [08:00<34:27,  4.30it/s]
 11%|█         | 1115/10000 [08:00<34:10,  4.33it/s]
 11%|█         | 1116/10000 [08:01<33:05,  4.47it/s]
 11%|█         | 1117/10000 [08:01<31:57,  4.63it/s]
 11%|█         | 1118/10000 [08:01<31:30,  4.70it/s]
 11%|█         | 1119/10000 [08:01<32:46,  4.52it/s]
 11%|█         | 1120/10000 [08:01<32:11,  4.60it/s]
                                                    
{'loss': 0.5091, 'grad_norm': 0.9407208561897278, 'learning_rate': 9.89561049676041e-05}

 11%|█         | 1120/10000 [08:01<32:11,  4.60it/s]
 11%|█         | 1121/10000 [08:02<31:59,  4.62it/s]
 11%|█         | 1122/10000 [08:02<31:32,  4.69it/s]
 11%|█         | 1123/10000 [08:02<31:26,  4.71it/s]
 11%|█         | 1124/10000 [08:02<30:59,  4.77it/s]
 11%|█▏        | 1125/10000 [08:02<30:32,  4.84it/s]
 11%|█▏        | 1126/10000 [08:03<30:42,  4.82it/s]
 11%|█▏        | 1127/10000 [08:03<30:12,  4.89it/s]
 11%|█▏        | 1128/10000 [08:03<29:41,  4.98it/s]
 11%|█▏        | 1129/10000 [08:03<30:58,  4.77it/s]
 11%|█▏        | 1130/10000 [08:03<31:49,  4.64it/s]
                                                    
{'loss': 0.4983, 'grad_norm': 0.9975925087928772, 'learning_rate': 9.89222267629528e-05}

 11%|█▏        | 1130/10000 [08:04<31:49,  4.64it/s]
 11%|█▏        | 1131/10000 [08:04<32:52,  4.50it/s]
 11%|█▏        | 1132/10000 [08:04<32:50,  4.50it/s]
 11%|█▏        | 1133/10000 [08:04<32:18,  4.57it/s]
 11%|█▏        | 1134/10000 [08:04<31:20,  4.71it/s]
 11%|█▏        | 1135/10000 [08:05<31:09,  4.74it/s]
 11%|█▏        | 1136/10000 [08:05<31:17,  4.72it/s]
 11%|█▏        | 1137/10000 [08:05<30:51,  4.79it/s]
 11%|█▏        | 1138/10000 [08:05<31:53,  4.63it/s]
 11%|█▏        | 1139/10000 [08:05<32:37,  4.53it/s]
 11%|█▏        | 1140/10000 [08:06<34:35,  4.27it/s]
                                                    
{'loss': 0.4989, 'grad_norm': 0.8524509072303772, 'learning_rate': 9.888781355266763e-05}

 11%|█▏        | 1140/10000 [08:06<34:35,  4.27it/s]
 11%|█▏        | 1141/10000 [08:06<36:12,  4.08it/s]
 11%|█▏        | 1142/10000 [08:06<36:39,  4.03it/s]
 11%|█▏        | 1143/10000 [08:06<35:46,  4.13it/s]
 11%|█▏        | 1144/10000 [08:07<33:57,  4.35it/s]
 11%|█▏        | 1145/10000 [08:07<32:42,  4.51it/s]
 11%|█▏        | 1146/10000 [08:07<32:36,  4.53it/s]
 11%|█▏        | 1147/10000 [08:07<34:19,  4.30it/s]
 11%|█▏        | 1148/10000 [08:08<38:17,  3.85it/s]
 11%|█▏        | 1149/10000 [08:08<40:38,  3.63it/s]
 12%|█▏        | 1150/10000 [08:08<40:32,  3.64it/s]
                                                    
{'loss': 0.5124, 'grad_norm': 0.9640136957168579, 'learning_rate': 9.885286571308598e-05}

 12%|█▏        | 1150/10000 [08:08<40:32,  3.64it/s]
 12%|█▏        | 1151/10000 [08:08<37:51,  3.90it/s]
 12%|█▏        | 1152/10000 [08:09<35:40,  4.13it/s]
 12%|█▏        | 1153/10000 [08:09<35:03,  4.21it/s]
 12%|█▏        | 1154/10000 [08:09<34:51,  4.23it/s]
 12%|█▏        | 1155/10000 [08:09<35:57,  4.10it/s]
 12%|█▏        | 1156/10000 [08:10<36:37,  4.02it/s]
 12%|█▏        | 1157/10000 [08:10<36:33,  4.03it/s]
 12%|█▏        | 1158/10000 [08:10<34:48,  4.23it/s]
 12%|█▏        | 1159/10000 [08:10<32:33,  4.53it/s]
 12%|█▏        | 1160/10000 [08:11<31:14,  4.72it/s]
                                                    
{'loss': 0.4945, 'grad_norm': 0.8477168083190918, 'learning_rate': 9.881738362639182e-05}

 12%|█▏        | 1160/10000 [08:11<31:14,  4.72it/s]
 12%|█▏        | 1161/10000 [08:11<30:46,  4.79it/s]
 12%|█▏        | 1162/10000 [08:11<31:14,  4.72it/s]
 12%|█▏        | 1163/10000 [08:11<34:30,  4.27it/s]
 12%|█▏        | 1164/10000 [08:11<35:08,  4.19it/s]
 12%|█▏        | 1165/10000 [08:12<35:26,  4.16it/s]
 12%|█▏        | 1166/10000 [08:12<35:17,  4.17it/s]
 12%|█▏        | 1167/10000 [08:12<34:21,  4.28it/s]
 12%|█▏        | 1168/10000 [08:12<33:08,  4.44it/s]
 12%|█▏        | 1169/10000 [08:13<31:27,  4.68it/s]
 12%|█▏        | 1170/10000 [08:13<32:16,  4.56it/s]
                                                    
{'loss': 0.4827, 'grad_norm': 1.033988356590271, 'learning_rate': 9.878136768061154e-05}

 12%|█▏        | 1170/10000 [08:13<32:16,  4.56it/s]
 12%|█▏        | 1171/10000 [08:13<33:16,  4.42it/s]
 12%|█▏        | 1172/10000 [08:13<32:50,  4.48it/s]
 12%|█▏        | 1173/10000 [08:13<33:08,  4.44it/s]
 12%|█▏        | 1174/10000 [08:14<32:10,  4.57it/s]
 12%|█▏        | 1175/10000 [08:14<31:11,  4.72it/s]
 12%|█▏        | 1176/10000 [08:14<30:54,  4.76it/s]
 12%|█▏        | 1177/10000 [08:14<30:28,  4.83it/s]
 12%|█▏        | 1178/10000 [08:14<30:21,  4.84it/s]
 12%|█▏        | 1179/10000 [08:15<31:20,  4.69it/s]
 12%|█▏        | 1180/10000 [08:15<31:18,  4.69it/s]
                                                    
{'loss': 0.478, 'grad_norm': 0.9084174633026123, 'learning_rate': 9.874481826960979e-05}

 12%|█▏        | 1180/10000 [08:15<31:18,  4.69it/s]
 12%|█▏        | 1181/10000 [08:15<31:32,  4.66it/s]
 12%|█▏        | 1182/10000 [08:15<31:19,  4.69it/s]
 12%|█▏        | 1183/10000 [08:16<30:47,  4.77it/s]
 12%|█▏        | 1184/10000 [08:16<30:19,  4.85it/s]
 12%|█▏        | 1185/10000 [08:16<30:01,  4.89it/s]
 12%|█▏        | 1186/10000 [08:16<30:10,  4.87it/s]
 12%|█▏        | 1187/10000 [08:16<30:31,  4.81it/s]
 12%|█▏        | 1188/10000 [08:17<32:02,  4.58it/s]
 12%|█▏        | 1189/10000 [08:17<33:53,  4.33it/s]
 12%|█▏        | 1190/10000 [08:17<35:11,  4.17it/s]
                                                    
{'loss': 0.4874, 'grad_norm': 0.8890127539634705, 'learning_rate': 9.870773579308503e-05}

 12%|█▏        | 1190/10000 [08:17<35:11,  4.17it/s]
 12%|█▏        | 1191/10000 [08:17<36:05,  4.07it/s]
 12%|█▏        | 1192/10000 [08:18<35:48,  4.10it/s]
 12%|█▏        | 1193/10000 [08:18<35:18,  4.16it/s]
 12%|█▏        | 1194/10000 [08:18<33:45,  4.35it/s]
 12%|█▏        | 1195/10000 [08:18<33:10,  4.42it/s]
 12%|█▏        | 1196/10000 [08:19<32:40,  4.49it/s]
 12%|█▏        | 1197/10000 [08:19<32:54,  4.46it/s]
 12%|█▏        | 1198/10000 [08:19<33:14,  4.41it/s]
 12%|█▏        | 1199/10000 [08:19<34:23,  4.27it/s]
 12%|█▏        | 1200/10000 [08:19<34:23,  4.26it/s]
                                                    
{'loss': 0.4648, 'grad_norm': 0.936534583568573, 'learning_rate': 9.867012065656533e-05}

 12%|█▏        | 1200/10000 [08:19<34:23,  4.26it/s]
 12%|█▏        | 1201/10000 [08:20<34:30,  4.25it/s]
 12%|█▏        | 1202/10000 [08:20<34:10,  4.29it/s]
 12%|█▏        | 1203/10000 [08:20<34:11,  4.29it/s]
 12%|█▏        | 1204/10000 [08:20<33:08,  4.42it/s]
 12%|█▏        | 1205/10000 [08:21<32:29,  4.51it/s]
 12%|█▏        | 1206/10000 [08:21<31:34,  4.64it/s]
 12%|█▏        | 1207/10000 [08:21<31:15,  4.69it/s]
 12%|█▏        | 1208/10000 [08:21<32:21,  4.53it/s]
 12%|█▏        | 1209/10000 [08:21<33:31,  4.37it/s]
 12%|█▏        | 1210/10000 [08:22<34:08,  4.29it/s]
                                                    
{'loss': 0.4771, 'grad_norm': 0.7788175344467163, 'learning_rate': 9.863197327140376e-05}

 12%|█▏        | 1210/10000 [08:22<34:08,  4.29it/s]
 12%|█▏        | 1211/10000 [08:22<34:05,  4.30it/s]
 12%|█▏        | 1212/10000 [08:22<34:09,  4.29it/s]
 12%|█▏        | 1213/10000 [08:22<32:58,  4.44it/s]
 12%|█▏        | 1214/10000 [08:23<32:28,  4.51it/s]
 12%|█▏        | 1215/10000 [08:23<32:23,  4.52it/s]
 12%|█▏        | 1216/10000 [08:23<32:36,  4.49it/s]
 12%|█▏        | 1217/10000 [08:23<33:23,  4.38it/s]
 12%|█▏        | 1218/10000 [08:24<34:47,  4.21it/s]
 12%|█▏        | 1219/10000 [08:24<35:41,  4.10it/s]
 12%|█▏        | 1220/10000 [08:24<35:25,  4.13it/s]
                                                    
{'loss': 0.4759, 'grad_norm': 0.7344397902488708, 'learning_rate': 9.859329405477403e-05}

 12%|█▏        | 1220/10000 [08:24<35:25,  4.13it/s]
 12%|█▏        | 1221/10000 [08:24<34:23,  4.25it/s]
 12%|█▏        | 1222/10000 [08:24<33:20,  4.39it/s]
 12%|█▏        | 1223/10000 [08:25<32:13,  4.54it/s]
 12%|█▏        | 1224/10000 [08:25<31:24,  4.66it/s]
 12%|█▏        | 1225/10000 [08:25<31:01,  4.71it/s]
 12%|█▏        | 1226/10000 [08:25<31:47,  4.60it/s]
 12%|█▏        | 1227/10000 [08:26<33:35,  4.35it/s]
 12%|█▏        | 1228/10000 [08:26<35:33,  4.11it/s]
 12%|█▏        | 1229/10000 [08:26<36:36,  3.99it/s]
 12%|█▏        | 1230/10000 [08:26<36:04,  4.05it/s]
                                                    
{'loss': 0.4582, 'grad_norm': 0.9831973910331726, 'learning_rate': 9.855408342966585e-05}

 12%|█▏        | 1230/10000 [08:26<36:04,  4.05it/s]
 12%|█▏        | 1231/10000 [08:27<34:14,  4.27it/s]
 12%|█▏        | 1232/10000 [08:27<34:03,  4.29it/s]
 12%|█▏        | 1233/10000 [08:27<34:29,  4.24it/s]
 12%|█▏        | 1234/10000 [08:27<34:23,  4.25it/s]
 12%|█▏        | 1235/10000 [08:28<36:11,  4.04it/s]
 12%|█▏        | 1236/10000 [08:28<37:12,  3.93it/s]
 12%|█▏        | 1237/10000 [08:28<36:24,  4.01it/s]
 12%|█▏        | 1238/10000 [08:28<35:30,  4.11it/s]
 12%|█▏        | 1239/10000 [08:28<34:05,  4.28it/s]
 12%|█▏        | 1240/10000 [08:29<33:00,  4.42it/s]
                                                    
{'loss': 0.4693, 'grad_norm': 0.7773613929748535, 'learning_rate': 9.851434182488033e-05}

 12%|█▏        | 1240/10000 [08:29<33:00,  4.42it/s]
 12%|█▏        | 1241/10000 [08:29<34:32,  4.23it/s]
 12%|█▏        | 1242/10000 [08:29<34:50,  4.19it/s]
 12%|█▏        | 1243/10000 [08:29<35:04,  4.16it/s]
 12%|█▏        | 1244/10000 [08:30<36:32,  3.99it/s]
 12%|█▏        | 1245/10000 [08:30<35:48,  4.08it/s]
 12%|█▏        | 1246/10000 [08:30<35:15,  4.14it/s]
 12%|█▏        | 1247/10000 [08:30<34:08,  4.27it/s]
 12%|█▏        | 1248/10000 [08:31<33:24,  4.37it/s]
 12%|█▏        | 1249/10000 [08:31<32:41,  4.46it/s]
 12%|█▎        | 1250/10000 [08:31<32:33,  4.48it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 5: Caching shard...

Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.32 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.58 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.69 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.75 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.95 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.97 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 1250: eval/loss = 0.4586 (36s)

                                                    
{'loss': 0.4781, 'grad_norm': 0.7989398241043091, 'learning_rate': 9.84740696750253e-05}

 12%|█▎        | 1250/10000 [09:07<32:33,  4.48it/s]
 13%|█▎        | 1251/10000 [09:07<26:48:56, 11.03s/it]
 13%|█▎        | 1252/10000 [09:07<18:53:52,  7.78s/it]
 13%|█▎        | 1253/10000 [09:08<13:21:31,  5.50s/it]
 13%|█▎        | 1254/10000 [09:08<9:29:18,  3.91s/it] 
 13%|█▎        | 1255/10000 [09:08<6:46:53,  2.79s/it]
 13%|█▎        | 1256/10000 [09:08<4:52:48,  2.01s/it]
 13%|█▎        | 1257/10000 [09:08<3:32:36,  1.46s/it]
 13%|█▎        | 1258/10000 [09:09<2:37:13,  1.08s/it]
 13%|█▎        | 1259/10000 [09:09<1:57:46,  1.24it/s]
 13%|█▎        | 1260/10000 [09:09<1:30:12,  1.61it/s]
                                                      
{'loss': 0.4516, 'grad_norm': 0.8139141201972961, 'learning_rate': 9.843326742051055e-05}

 13%|█▎        | 1260/10000 [09:09<1:30:12,  1.61it/s]
 13%|█▎        | 1261/10000 [09:09<1:12:59,  2.00it/s]
 13%|█▎        | 1262/10000 [09:09<58:56,  2.47it/s]  
 13%|█▎        | 1263/10000 [09:10<49:01,  2.97it/s]
 13%|█▎        | 1264/10000 [09:10<42:16,  3.44it/s]
 13%|█▎        | 1265/10000 [09:10<37:43,  3.86it/s]
 13%|█▎        | 1266/10000 [09:10<34:28,  4.22it/s]
 13%|█▎        | 1267/10000 [09:10<32:27,  4.48it/s]
 13%|█▎        | 1268/10000 [09:10<30:32,  4.77it/s]
 13%|█▎        | 1269/10000 [09:11<29:09,  4.99it/s]
 13%|█▎        | 1270/10000 [09:11<28:05,  5.18it/s]
                                                    
{'loss': 0.4651, 'grad_norm': 0.9229570031166077, 'learning_rate': 9.839193550754297e-05}

 13%|█▎        | 1270/10000 [09:11<28:05,  5.18it/s]
 13%|█▎        | 1271/10000 [09:11<27:57,  5.20it/s]
 13%|█▎        | 1272/10000 [09:11<27:09,  5.36it/s]
 13%|█▎        | 1273/10000 [09:11<29:30,  4.93it/s]
 13%|█▎        | 1274/10000 [09:12<29:22,  4.95it/s]
 13%|█▎        | 1275/10000 [09:12<30:28,  4.77it/s]
 13%|█▎        | 1276/10000 [09:12<31:25,  4.63it/s]
 13%|█▎        | 1277/10000 [09:12<29:37,  4.91it/s]
 13%|█▎        | 1278/10000 [09:12<28:30,  5.10it/s]
 13%|█▎        | 1279/10000 [09:13<27:30,  5.28it/s]
 13%|█▎        | 1280/10000 [09:13<26:46,  5.43it/s]
                                                    
{'loss': 0.4733, 'grad_norm': 1.2144124507904053, 'learning_rate': 9.835007438812177e-05}

 13%|█▎        | 1280/10000 [09:13<26:46,  5.43it/s]
 13%|█▎        | 1281/10000 [09:13<26:56,  5.40it/s]
 13%|█▎        | 1282/10000 [09:13<26:33,  5.47it/s]
 13%|█▎        | 1283/10000 [09:13<26:13,  5.54it/s]
 13%|█▎        | 1284/10000 [09:14<26:07,  5.56it/s]
 13%|█▎        | 1285/10000 [09:14<26:32,  5.47it/s]
 13%|█▎        | 1286/10000 [09:14<26:21,  5.51it/s]
 13%|█▎        | 1287/10000 [09:14<26:10,  5.55it/s]
 13%|█▎        | 1288/10000 [09:14<26:45,  5.43it/s]
 13%|█▎        | 1289/10000 [09:14<26:51,  5.41it/s]
 13%|█▎        | 1290/10000 [09:15<26:53,  5.40it/s]
                                                    
{'loss': 0.4698, 'grad_norm': 0.951970100402832, 'learning_rate': 9.830768452003341e-05}

 13%|█▎        | 1290/10000 [09:15<26:53,  5.40it/s]
 13%|█▎        | 1291/10000 [09:15<27:30,  5.28it/s]
 13%|█▎        | 1292/10000 [09:15<27:27,  5.28it/s]
 13%|█▎        | 1293/10000 [09:15<27:15,  5.32it/s]
 13%|█▎        | 1294/10000 [09:15<26:57,  5.38it/s]
 13%|█▎        | 1295/10000 [09:16<26:53,  5.39it/s]
 13%|█▎        | 1296/10000 [09:16<26:47,  5.41it/s]
 13%|█▎        | 1297/10000 [09:16<26:41,  5.43it/s]
 13%|█▎        | 1298/10000 [09:16<26:55,  5.39it/s]
 13%|█▎        | 1299/10000 [09:16<26:42,  5.43it/s]
 13%|█▎        | 1300/10000 [09:16<27:04,  5.36it/s]
                                                    
{'loss': 0.4578, 'grad_norm': 1.0694864988327026, 'learning_rate': 9.826476636684671e-05}

 13%|█▎        | 1300/10000 [09:17<27:04,  5.36it/s]
 13%|█▎        | 1301/10000 [09:17<27:27,  5.28it/s]
 13%|█▎        | 1302/10000 [09:17<27:10,  5.34it/s]
 13%|█▎        | 1303/10000 [09:17<27:07,  5.34it/s]
 13%|█▎        | 1304/10000 [09:17<27:04,  5.35it/s]
 13%|█▎        | 1305/10000 [09:17<27:39,  5.24it/s]
 13%|█▎        | 1306/10000 [09:18<28:20,  5.11it/s]
 13%|█▎        | 1307/10000 [09:18<29:11,  4.96it/s]
 13%|█▎        | 1308/10000 [09:18<29:13,  4.96it/s]
 13%|█▎        | 1309/10000 [09:18<29:13,  4.96it/s]
 13%|█▎        | 1310/10000 [09:18<28:07,  5.15it/s]
                                                    
{'loss': 0.466, 'grad_norm': 0.8484354019165039, 'learning_rate': 9.822132039790773e-05}

 13%|█▎        | 1310/10000 [09:18<28:07,  5.15it/s]
 13%|█▎        | 1311/10000 [09:19<27:41,  5.23it/s]
 13%|█▎        | 1312/10000 [09:19<27:02,  5.35it/s]
 13%|█▎        | 1313/10000 [09:19<26:41,  5.43it/s]
 13%|█▎        | 1314/10000 [09:19<26:57,  5.37it/s]
 13%|█▎        | 1315/10000 [09:19<27:26,  5.28it/s]
 13%|█▎        | 1316/10000 [09:20<28:13,  5.13it/s]
 13%|█▎        | 1317/10000 [09:20<29:05,  4.97it/s]
 13%|█▎        | 1318/10000 [09:20<29:18,  4.94it/s]
 13%|█▎        | 1319/10000 [09:20<28:13,  5.13it/s]
 13%|█▎        | 1320/10000 [09:20<27:29,  5.26it/s]
                                                    
{'loss': 0.4316, 'grad_norm': 0.8231856226921082, 'learning_rate': 9.817734708833461e-05}

 13%|█▎        | 1320/10000 [09:20<27:29,  5.26it/s]
 13%|█▎        | 1321/10000 [09:21<27:25,  5.28it/s]
 13%|█▎        | 1322/10000 [09:21<28:00,  5.16it/s]
 13%|█▎        | 1323/10000 [09:21<28:20,  5.10it/s]
 13%|█▎        | 1324/10000 [09:21<28:34,  5.06it/s]
 13%|█▎        | 1325/10000 [09:21<28:54,  5.00it/s]
 13%|█▎        | 1326/10000 [09:22<28:22,  5.09it/s]
 13%|█▎        | 1327/10000 [09:22<27:32,  5.25it/s]
 13%|█▎        | 1328/10000 [09:22<26:51,  5.38it/s]
 13%|█▎        | 1329/10000 [09:22<26:41,  5.42it/s]
 13%|█▎        | 1330/10000 [09:22<26:35,  5.43it/s]
                                                    
{'loss': 0.4522, 'grad_norm': 0.9760980606079102, 'learning_rate': 9.813284691901243e-05}

 13%|█▎        | 1330/10000 [09:22<26:35,  5.43it/s]
 13%|█▎        | 1331/10000 [09:22<27:02,  5.34it/s]
 13%|█▎        | 1332/10000 [09:23<26:59,  5.35it/s]
 13%|█▎        | 1333/10000 [09:23<26:58,  5.35it/s]
 13%|█▎        | 1334/10000 [09:23<26:27,  5.46it/s]
 13%|█▎        | 1335/10000 [09:23<26:11,  5.51it/s]
 13%|█▎        | 1336/10000 [09:23<26:38,  5.42it/s]
 13%|█▎        | 1337/10000 [09:24<27:43,  5.21it/s]
 13%|█▎        | 1338/10000 [09:24<29:22,  4.91it/s]
 13%|█▎        | 1339/10000 [09:24<30:22,  4.75it/s]
 13%|█▎        | 1340/10000 [09:24<30:30,  4.73it/s]
                                                    
{'loss': 0.453, 'grad_norm': 0.9553073644638062, 'learning_rate': 9.808782037658792e-05}

 13%|█▎        | 1340/10000 [09:24<30:30,  4.73it/s]
 13%|█▎        | 1341/10000 [09:24<31:36,  4.57it/s]
 13%|█▎        | 1342/10000 [09:25<31:29,  4.58it/s]
 13%|█▎        | 1343/10000 [09:25<30:56,  4.66it/s]
 13%|█▎        | 1344/10000 [09:25<31:04,  4.64it/s]
 13%|█▎        | 1345/10000 [09:25<31:15,  4.62it/s]
 13%|█▎        | 1346/10000 [09:26<31:26,  4.59it/s]
 13%|█▎        | 1347/10000 [09:26<32:11,  4.48it/s]
 13%|█▎        | 1348/10000 [09:26<31:21,  4.60it/s]
 13%|█▎        | 1349/10000 [09:26<30:19,  4.75it/s]
 14%|█▎        | 1350/10000 [09:26<29:01,  4.97it/s]
                                                    
{'loss': 0.427, 'grad_norm': 0.9428948760032654, 'learning_rate': 9.804226795346411e-05}

 14%|█▎        | 1350/10000 [09:26<29:01,  4.97it/s]
 14%|█▎        | 1351/10000 [09:27<28:25,  5.07it/s]
 14%|█▎        | 1352/10000 [09:27<28:02,  5.14it/s]
 14%|█▎        | 1353/10000 [09:27<28:03,  5.14it/s]
 14%|█▎        | 1354/10000 [09:27<28:19,  5.09it/s]
 14%|█▎        | 1355/10000 [09:27<28:43,  5.02it/s]
 14%|█▎        | 1356/10000 [09:28<28:31,  5.05it/s]
 14%|█▎        | 1357/10000 [09:28<27:42,  5.20it/s]
 14%|█▎        | 1358/10000 [09:28<27:12,  5.29it/s]
 14%|█▎        | 1359/10000 [09:28<26:40,  5.40it/s]
 14%|█▎        | 1360/10000 [09:28<26:14,  5.49it/s]
                                                    
{'loss': 0.4177, 'grad_norm': 0.9390555620193481, 'learning_rate': 9.799619014779503e-05}

 14%|█▎        | 1360/10000 [09:28<26:14,  5.49it/s]
 14%|█▎        | 1361/10000 [09:28<26:59,  5.34it/s]
 14%|█▎        | 1362/10000 [09:29<27:18,  5.27it/s]
 14%|█▎        | 1363/10000 [09:29<27:35,  5.22it/s]
 14%|█▎        | 1364/10000 [09:29<27:49,  5.17it/s]
 14%|█▎        | 1365/10000 [09:29<27:35,  5.21it/s]
 14%|█▎        | 1366/10000 [09:29<26:52,  5.35it/s]
 14%|█▎        | 1367/10000 [09:30<26:25,  5.44it/s]
 14%|█▎        | 1368/10000 [09:30<26:06,  5.51it/s]
 14%|█▎        | 1369/10000 [09:30<25:57,  5.54it/s]
 14%|█▎        | 1370/10000 [09:30<26:17,  5.47it/s]
                                                    
{'loss': 0.4445, 'grad_norm': 1.0052635669708252, 'learning_rate': 9.794958746348013e-05}

 14%|█▎        | 1370/10000 [09:30<26:17,  5.47it/s]
 14%|█▎        | 1371/10000 [09:30<27:47,  5.17it/s]
 14%|█▎        | 1372/10000 [09:31<27:58,  5.14it/s]
 14%|█▎        | 1373/10000 [09:31<27:46,  5.18it/s]
 14%|█▎        | 1374/10000 [09:31<27:44,  5.18it/s]
 14%|█▍        | 1375/10000 [09:31<26:50,  5.36it/s]
 14%|█▍        | 1376/10000 [09:31<27:01,  5.32it/s]
 14%|█▍        | 1377/10000 [09:31<26:26,  5.44it/s]
 14%|█▍        | 1378/10000 [09:32<26:03,  5.51it/s]
 14%|█▍        | 1379/10000 [09:32<26:19,  5.46it/s]
 14%|█▍        | 1380/10000 [09:32<26:22,  5.45it/s]
                                                    
{'loss': 0.439, 'grad_norm': 0.8766794800758362, 'learning_rate': 9.790246041015896e-05}

 14%|█▍        | 1380/10000 [09:32<26:22,  5.45it/s]
 14%|█▍        | 1381/10000 [09:32<26:58,  5.33it/s]
 14%|█▍        | 1382/10000 [09:32<27:07,  5.29it/s]
 14%|█▍        | 1383/10000 [09:33<26:57,  5.33it/s]
 14%|█▍        | 1384/10000 [09:33<26:58,  5.32it/s]
 14%|█▍        | 1385/10000 [09:33<26:34,  5.40it/s]
 14%|█▍        | 1386/10000 [09:33<26:17,  5.46it/s]
 14%|█▍        | 1387/10000 [09:33<26:30,  5.42it/s]
 14%|█▍        | 1388/10000 [09:33<26:02,  5.51it/s]
 14%|█▍        | 1389/10000 [09:34<25:56,  5.53it/s]
 14%|█▍        | 1390/10000 [09:34<25:45,  5.57it/s]
                                                    
{'loss': 0.4313, 'grad_norm': 1.072033405303955, 'learning_rate': 9.785480950320538e-05}

 14%|█▍        | 1390/10000 [09:34<25:45,  5.57it/s]
 14%|█▍        | 1391/10000 [09:34<26:12,  5.48it/s]
 14%|█▍        | 1392/10000 [09:34<25:52,  5.54it/s]
 14%|█▍        | 1393/10000 [09:34<25:46,  5.57it/s]
 14%|█▍        | 1394/10000 [09:35<26:06,  5.50it/s]
 14%|█▍        | 1395/10000 [09:35<25:54,  5.54it/s]
 14%|█▍        | 1396/10000 [09:35<26:15,  5.46it/s]
 14%|█▍        | 1397/10000 [09:35<26:35,  5.39it/s]
 14%|█▍        | 1398/10000 [09:35<26:39,  5.38it/s]
 14%|█▍        | 1399/10000 [09:35<26:23,  5.43it/s]
 14%|█▍        | 1400/10000 [09:36<26:11,  5.47it/s]
                                                    
{'loss': 0.4323, 'grad_norm': 0.9433143734931946, 'learning_rate': 9.78066352637221e-05}

 14%|█▍        | 1400/10000 [09:36<26:11,  5.47it/s]
 14%|█▍        | 1401/10000 [09:36<26:17,  5.45it/s]
 14%|█▍        | 1402/10000 [09:36<25:58,  5.52it/s]
 14%|█▍        | 1403/10000 [09:36<25:43,  5.57it/s]
 14%|█▍        | 1404/10000 [09:36<25:50,  5.54it/s]
 14%|█▍        | 1405/10000 [09:37<25:36,  5.59it/s]
 14%|█▍        | 1406/10000 [09:37<25:32,  5.61it/s]
 14%|█▍        | 1407/10000 [09:37<25:30,  5.62it/s]
 14%|█▍        | 1408/10000 [09:37<25:34,  5.60it/s]
 14%|█▍        | 1409/10000 [09:37<26:04,  5.49it/s]
 14%|█▍        | 1410/10000 [09:37<26:10,  5.47it/s]
                                                    
{'loss': 0.4224, 'grad_norm': 0.9185171723365784, 'learning_rate': 9.775793821853488e-05}

 14%|█▍        | 1410/10000 [09:38<26:10,  5.47it/s]
 14%|█▍        | 1411/10000 [09:38<26:34,  5.39it/s]
 14%|█▍        | 1412/10000 [09:38<26:17,  5.44it/s]
 14%|█▍        | 1413/10000 [09:38<27:11,  5.26it/s]
 14%|█▍        | 1414/10000 [09:38<27:27,  5.21it/s]
 14%|█▍        | 1415/10000 [09:38<26:59,  5.30it/s]
 14%|█▍        | 1416/10000 [09:39<26:27,  5.41it/s]
 14%|█▍        | 1417/10000 [09:39<25:58,  5.51it/s]
 14%|█▍        | 1418/10000 [09:39<25:43,  5.56it/s]
 14%|█▍        | 1419/10000 [09:39<26:04,  5.49it/s]
 14%|█▍        | 1420/10000 [09:39<26:08,  5.47it/s]
                                                    
{'loss': 0.4136, 'grad_norm': 1.4804445505142212, 'learning_rate': 9.77087189001868e-05}

 14%|█▍        | 1420/10000 [09:39<26:08,  5.47it/s]
 14%|█▍        | 1421/10000 [09:40<26:34,  5.38it/s]
 14%|█▍        | 1422/10000 [09:40<26:16,  5.44it/s]
 14%|█▍        | 1423/10000 [09:40<25:45,  5.55it/s]
 14%|█▍        | 1424/10000 [09:40<25:32,  5.60it/s]
 14%|█▍        | 1425/10000 [09:40<25:24,  5.63it/s]
 14%|█▍        | 1426/10000 [09:41<31:20,  4.56it/s]
 14%|█▍        | 1427/10000 [09:41<30:24,  4.70it/s]
 14%|█▍        | 1428/10000 [09:41<29:43,  4.81it/s]
 14%|█▍        | 1429/10000 [09:41<29:18,  4.87it/s]
 14%|█▍        | 1430/10000 [09:41<28:30,  5.01it/s]
                                                    
{'loss': 0.4152, 'grad_norm': 0.8452776670455933, 'learning_rate': 9.765897784693243e-05}

 14%|█▍        | 1430/10000 [09:41<28:30,  5.01it/s]
 14%|█▍        | 1431/10000 [09:42<27:51,  5.13it/s]
 14%|█▍        | 1432/10000 [09:42<26:57,  5.30it/s]
 14%|█▍        | 1433/10000 [09:42<26:23,  5.41it/s]
 14%|█▍        | 1434/10000 [09:42<26:26,  5.40it/s]
 14%|█▍        | 1435/10000 [09:42<26:25,  5.40it/s]
 14%|█▍        | 1436/10000 [09:42<26:29,  5.39it/s]
 14%|█▍        | 1437/10000 [09:43<26:23,  5.41it/s]
 14%|█▍        | 1438/10000 [09:43<26:17,  5.43it/s]
 14%|█▍        | 1439/10000 [09:43<26:31,  5.38it/s]
 14%|█▍        | 1440/10000 [09:43<26:02,  5.48it/s]
                                                    
{'loss': 0.4121, 'grad_norm': 0.9437452554702759, 'learning_rate': 9.760871560273197e-05}

 14%|█▍        | 1440/10000 [09:43<26:02,  5.48it/s]
 14%|█▍        | 1441/10000 [09:43<26:12,  5.44it/s]
 14%|█▍        | 1442/10000 [09:44<25:53,  5.51it/s]
 14%|█▍        | 1443/10000 [09:44<25:45,  5.54it/s]
 14%|█▍        | 1444/10000 [09:44<25:42,  5.55it/s]
 14%|█▍        | 1445/10000 [09:44<26:02,  5.47it/s]
 14%|█▍        | 1446/10000 [09:44<26:04,  5.47it/s]
 14%|█▍        | 1447/10000 [09:44<26:22,  5.40it/s]
 14%|█▍        | 1448/10000 [09:45<26:35,  5.36it/s]
 14%|█▍        | 1449/10000 [09:45<27:11,  5.24it/s]
 14%|█▍        | 1450/10000 [09:45<27:34,  5.17it/s]
                                                    
{'loss': 0.4032, 'grad_norm': 0.9407142400741577, 'learning_rate': 9.755793271724526e-05}

 14%|█▍        | 1450/10000 [09:45<27:34,  5.17it/s]
 15%|█▍        | 1451/10000 [09:45<27:53,  5.11it/s]
 15%|█▍        | 1452/10000 [09:45<27:17,  5.22it/s]
 15%|█▍        | 1453/10000 [09:46<26:53,  5.30it/s]
 15%|█▍        | 1454/10000 [09:46<26:53,  5.30it/s]
 15%|█▍        | 1455/10000 [09:46<26:59,  5.28it/s]
 15%|█▍        | 1456/10000 [09:46<27:39,  5.15it/s]Rank 0, Worker 4: Wait for shard 30 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 15%|█▍        | 1457/10000 [09:46<28:02,  5.08it/s]
 15%|█▍        | 1458/10000 [09:47<28:00,  5.08it/s]
 15%|█▍        | 1459/10000 [09:47<28:11,  5.05it/s]
 15%|█▍        | 1460/10000 [09:47<28:34,  4.98it/s]
                                                    
{'loss': 0.418, 'grad_norm': 1.034866213798523, 'learning_rate': 9.750662974582584e-05}

 15%|█▍        | 1460/10000 [09:47<28:34,  4.98it/s]
 15%|█▍        | 1461/10000 [09:47<28:43,  4.95it/s]
 15%|█▍        | 1462/10000 [09:47<28:13,  5.04it/s]
 15%|█▍        | 1463/10000 [09:48<29:40,  4.79it/s]
 15%|█▍        | 1464/10000 [09:48<29:39,  4.80it/s]
 15%|█▍        | 1465/10000 [09:48<29:34,  4.81it/s]
 15%|█▍        | 1466/10000 [09:48<29:08,  4.88it/s]
 15%|█▍        | 1467/10000 [09:48<28:28,  4.99it/s]
 15%|█▍        | 1468/10000 [09:49<27:36,  5.15it/s]
 15%|█▍        | 1469/10000 [09:49<27:11,  5.23it/s]
 15%|█▍        | 1470/10000 [09:49<26:47,  5.31it/s]
                                                    
{'loss': 0.4, 'grad_norm': 0.8009440302848816, 'learning_rate': 9.745480724951473e-05}

 15%|█▍        | 1470/10000 [09:49<26:47,  5.31it/s]
 15%|█▍        | 1471/10000 [09:49<28:51,  4.93it/s]
 15%|█▍        | 1472/10000 [09:49<30:56,  4.59it/s]
 15%|█▍        | 1473/10000 [09:50<30:58,  4.59it/s]
 15%|█▍        | 1474/10000 [09:50<30:30,  4.66it/s]
 15%|█▍        | 1475/10000 [09:50<30:02,  4.73it/s]
 15%|█▍        | 1476/10000 [09:50<28:32,  4.98it/s]
 15%|█▍        | 1477/10000 [09:50<27:44,  5.12it/s]
 15%|█▍        | 1478/10000 [09:51<27:04,  5.24it/s]
 15%|█▍        | 1479/10000 [09:51<27:12,  5.22it/s]
 15%|█▍        | 1480/10000 [09:51<27:49,  5.10it/s]
                                                    
{'loss': 0.3896, 'grad_norm': 0.9475518465042114, 'learning_rate': 9.740246579503447e-05}

 15%|█▍        | 1480/10000 [09:51<27:49,  5.10it/s]
 15%|█▍        | 1481/10000 [09:51<29:09,  4.87it/s]
 15%|█▍        | 1482/10000 [09:51<29:32,  4.81it/s]
 15%|█▍        | 1483/10000 [09:52<29:47,  4.76it/s]
 15%|█▍        | 1484/10000 [09:52<29:19,  4.84it/s]
 15%|█▍        | 1485/10000 [09:52<29:00,  4.89it/s]
 15%|█▍        | 1486/10000 [09:52<28:33,  4.97it/s]
 15%|█▍        | 1487/10000 [09:52<27:51,  5.09it/s]
 15%|█▍        | 1488/10000 [09:53<27:17,  5.20it/s]
 15%|█▍        | 1489/10000 [09:53<27:26,  5.17it/s]
 15%|█▍        | 1490/10000 [09:53<28:01,  5.06it/s]
                                                    
{'loss': 0.3797, 'grad_norm': 0.9906937479972839, 'learning_rate': 9.734960595478284e-05}

 15%|█▍        | 1490/10000 [09:53<28:01,  5.06it/s]
 15%|█▍        | 1491/10000 [09:53<28:42,  4.94it/s]
 15%|█▍        | 1492/10000 [09:53<28:47,  4.93it/s]
 15%|█▍        | 1493/10000 [09:54<28:04,  5.05it/s]Rank 0, Worker 5: Wait for shard 63 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 15%|█▍        | 1494/10000 [09:54<28:27,  4.98it/s]
 15%|█▍        | 1495/10000 [09:54<28:39,  4.95it/s]
 15%|█▍        | 1496/10000 [09:54<29:28,  4.81it/s]
 15%|█▍        | 1497/10000 [09:55<30:29,  4.65it/s]
 15%|█▍        | 1498/10000 [09:55<30:13,  4.69it/s]
 15%|█▍        | 1499/10000 [09:55<31:13,  4.54it/s]
 15%|█▌        | 1500/10000 [09:55<31:44,  4.46it/s]Rank 0, Worker 1: Caching shard...Rank 0, Worker 0: Caching shard...

Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.44 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.05 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.12 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.25 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.27 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.43 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 1500: eval/loss = 0.3840 (39s)

                                                    
{'loss': 0.4033, 'grad_norm': 0.9057230949401855, 'learning_rate': 9.729622830682657e-05}

 15%|█▌        | 1500/10000 [10:34<31:44,  4.46it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-1500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-1500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-1500/wandb_config.json

 15%|█▌        | 1501/10000 [11:01<47:16:06, 20.02s/it]
 15%|█▌        | 1502/10000 [11:02<33:12:52, 14.07s/it]
 15%|█▌        | 1503/10000 [11:02<23:22:48,  9.91s/it]
 15%|█▌        | 1504/10000 [11:02<16:29:15,  6.99s/it]
 15%|█▌        | 1505/10000 [11:02<11:39:46,  4.94s/it]
 15%|█▌        | 1506/10000 [11:02<8:17:18,  3.51s/it] 
 15%|█▌        | 1507/10000 [11:02<5:55:31,  2.51s/it]
 15%|█▌        | 1508/10000 [11:03<4:16:04,  1.81s/it]
 15%|█▌        | 1509/10000 [11:03<3:06:42,  1.32s/it]
 15%|█▌        | 1510/10000 [11:03<2:18:06,  1.02it/s]
                                                      
{'loss': 0.3957, 'grad_norm': 1.2511104345321655, 'learning_rate': 9.724233343489504e-05}

 15%|█▌        | 1510/10000 [11:03<2:18:06,  1.02it/s]
 15%|█▌        | 1511/10000 [11:03<1:44:19,  1.36it/s]
 15%|█▌        | 1512/10000 [11:03<1:20:26,  1.76it/s]
 15%|█▌        | 1513/10000 [11:04<1:03:51,  2.22it/s]
 15%|█▌        | 1514/10000 [11:04<52:40,  2.68it/s]  
 15%|█▌        | 1515/10000 [11:04<44:55,  3.15it/s]Rank 0, Worker 3: Wait for shard 29 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 15%|█▌        | 1516/10000 [11:04<39:57,  3.54it/s]
 15%|█▌        | 1517/10000 [11:04<36:03,  3.92it/s]
 15%|█▌        | 1518/10000 [11:04<32:51,  4.30it/s]
 15%|█▌        | 1519/10000 [11:05<31:19,  4.51it/s]Rank 0, Worker 1: Wait for shard 28 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 15%|█▌        | 1520/10000 [11:05<30:18,  4.66it/s]
                                                    
{'loss': 0.4016, 'grad_norm': 0.9278829097747803, 'learning_rate': 9.718792192837396e-05}

 15%|█▌        | 1520/10000 [11:05<30:18,  4.66it/s]
 15%|█▌        | 1521/10000 [11:05<30:00,  4.71it/s]
 15%|█▌        | 1522/10000 [11:05<29:57,  4.72it/s]
 15%|█▌        | 1523/10000 [11:05<29:39,  4.76it/s]
 15%|█▌        | 1524/10000 [11:06<29:35,  4.77it/s]
 15%|█▌        | 1525/10000 [11:06<29:47,  4.74it/s]
 15%|█▌        | 1526/10000 [11:06<28:49,  4.90it/s]
 15%|█▌        | 1527/10000 [11:06<28:06,  5.03it/s]
 15%|█▌        | 1528/10000 [11:06<27:41,  5.10it/s]
 15%|█▌        | 1529/10000 [11:07<27:36,  5.11it/s]
 15%|█▌        | 1530/10000 [11:07<27:56,  5.05it/s]
                                                    
{'loss': 0.3875, 'grad_norm': 0.893507719039917, 'learning_rate': 9.713299438229886e-05}

 15%|█▌        | 1530/10000 [11:07<27:56,  5.05it/s]
 15%|█▌        | 1531/10000 [11:07<28:30,  4.95it/s]
 15%|█▌        | 1532/10000 [11:07<28:30,  4.95it/s]
 15%|█▌        | 1533/10000 [11:07<28:49,  4.90it/s]
 15%|█▌        | 1534/10000 [11:08<28:52,  4.89it/s]
 15%|█▌        | 1535/10000 [11:08<28:14,  4.99it/s]
 15%|█▌        | 1536/10000 [11:08<28:56,  4.87it/s]
 15%|█▌        | 1537/10000 [11:08<28:58,  4.87it/s]
 15%|█▌        | 1538/10000 [11:09<28:46,  4.90it/s]
 15%|█▌        | 1539/10000 [11:09<29:47,  4.73it/s]
 15%|█▌        | 1540/10000 [11:09<29:57,  4.71it/s]
                                                    
{'loss': 0.3942, 'grad_norm': 0.9839492440223694, 'learning_rate': 9.707755139734855e-05}

 15%|█▌        | 1540/10000 [11:09<29:57,  4.71it/s]
 15%|█▌        | 1541/10000 [11:09<30:25,  4.63it/s]
 15%|█▌        | 1542/10000 [11:09<30:39,  4.60it/s]
 15%|█▌        | 1543/10000 [11:10<30:25,  4.63it/s]
 15%|█▌        | 1544/10000 [11:10<31:01,  4.54it/s]
 15%|█▌        | 1545/10000 [11:10<29:52,  4.72it/s]
 15%|█▌        | 1546/10000 [11:10<29:00,  4.86it/s]
 15%|█▌        | 1547/10000 [11:10<28:45,  4.90it/s]
 15%|█▌        | 1548/10000 [11:11<28:46,  4.90it/s]
 15%|█▌        | 1549/10000 [11:11<29:18,  4.81it/s]
 16%|█▌        | 1550/10000 [11:11<31:44,  4.44it/s]
                                                    
{'loss': 0.4019, 'grad_norm': 0.9847416877746582, 'learning_rate': 9.702159357983866e-05}

 16%|█▌        | 1550/10000 [11:11<31:44,  4.44it/s]
 16%|█▌        | 1551/10000 [11:11<31:26,  4.48it/s]
 16%|█▌        | 1552/10000 [11:12<30:33,  4.61it/s]
 16%|█▌        | 1553/10000 [11:12<30:00,  4.69it/s]
 16%|█▌        | 1554/10000 [11:12<29:15,  4.81it/s]
 16%|█▌        | 1555/10000 [11:12<29:26,  4.78it/s]
 16%|█▌        | 1556/10000 [11:12<29:55,  4.70it/s]
 16%|█▌        | 1557/10000 [11:13<30:12,  4.66it/s]
 16%|█▌        | 1558/10000 [11:13<31:05,  4.53it/s]
 16%|█▌        | 1559/10000 [11:13<31:38,  4.45it/s]
 16%|█▌        | 1560/10000 [11:13<31:06,  4.52it/s]
                                                    
{'loss': 0.392, 'grad_norm': 0.9709538817405701, 'learning_rate': 9.696512154171492e-05}

 16%|█▌        | 1560/10000 [11:13<31:06,  4.52it/s]
 16%|█▌        | 1561/10000 [11:14<31:27,  4.47it/s]
 16%|█▌        | 1562/10000 [11:14<30:17,  4.64it/s]
 16%|█▌        | 1563/10000 [11:14<29:44,  4.73it/s]
 16%|█▌        | 1564/10000 [11:14<29:20,  4.79it/s]
 16%|█▌        | 1565/10000 [11:14<29:01,  4.84it/s]
 16%|█▌        | 1566/10000 [11:15<28:56,  4.86it/s]Rank 0, Worker 0: Wait for shard 39 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 16%|█▌        | 1567/10000 [11:15<29:18,  4.80it/s]
 16%|█▌        | 1568/10000 [11:15<29:30,  4.76it/s]
 16%|█▌        | 1569/10000 [11:15<29:56,  4.69it/s]
 16%|█▌        | 1570/10000 [11:15<30:44,  4.57it/s]
                                                    
{'loss': 0.364, 'grad_norm': 0.856467068195343, 'learning_rate': 9.690813590054645e-05}

 16%|█▌        | 1570/10000 [11:15<30:44,  4.57it/s]
 16%|█▌        | 1571/10000 [11:16<31:58,  4.39it/s]
 16%|█▌        | 1572/10000 [11:16<31:37,  4.44it/s]
 16%|█▌        | 1573/10000 [11:16<30:51,  4.55it/s]
 16%|█▌        | 1574/10000 [11:16<31:02,  4.53it/s]
 16%|█▌        | 1575/10000 [11:17<30:48,  4.56it/s]
 16%|█▌        | 1576/10000 [11:17<30:22,  4.62it/s]
 16%|█▌        | 1577/10000 [11:17<30:29,  4.60it/s]
 16%|█▌        | 1578/10000 [11:17<30:09,  4.65it/s]
 16%|█▌        | 1579/10000 [11:17<29:54,  4.69it/s]
 16%|█▌        | 1580/10000 [11:18<29:45,  4.72it/s]
                                                    
{'loss': 0.3697, 'grad_norm': 0.8411120772361755, 'learning_rate': 9.685063727951914e-05}

 16%|█▌        | 1580/10000 [11:18<29:45,  4.72it/s]
 16%|█▌        | 1581/10000 [11:18<30:17,  4.63it/s]
 16%|█▌        | 1582/10000 [11:18<30:08,  4.65it/s]
 16%|█▌        | 1583/10000 [11:18<30:30,  4.60it/s]
 16%|█▌        | 1584/10000 [11:18<31:15,  4.49it/s]
 16%|█▌        | 1585/10000 [11:19<30:46,  4.56it/s]
 16%|█▌        | 1586/10000 [11:19<30:49,  4.55it/s]
 16%|█▌        | 1587/10000 [11:19<31:00,  4.52it/s]
 16%|█▌        | 1588/10000 [11:19<30:57,  4.53it/s]
 16%|█▌        | 1589/10000 [11:20<30:40,  4.57it/s]
 16%|█▌        | 1590/10000 [11:20<30:26,  4.61it/s]
                                                    
{'loss': 0.3483, 'grad_norm': 0.7915524244308472, 'learning_rate': 9.679262630742865e-05}

 16%|█▌        | 1590/10000 [11:20<30:26,  4.61it/s]
 16%|█▌        | 1591/10000 [11:20<30:29,  4.60it/s]
 16%|█▌        | 1592/10000 [11:20<29:12,  4.80it/s]
 16%|█▌        | 1593/10000 [11:20<27:57,  5.01it/s]
 16%|█▌        | 1594/10000 [11:21<27:49,  5.04it/s]
 16%|█▌        | 1595/10000 [11:21<27:46,  5.04it/s]
 16%|█▌        | 1596/10000 [11:21<28:58,  4.83it/s]
 16%|█▌        | 1597/10000 [11:21<30:37,  4.57it/s]
 16%|█▌        | 1598/10000 [11:21<30:36,  4.58it/s]
 16%|█▌        | 1599/10000 [11:22<30:43,  4.56it/s]
 16%|█▌        | 1600/10000 [11:22<30:35,  4.58it/s]
                                                    
{'loss': 0.3561, 'grad_norm': 1.0969499349594116, 'learning_rate': 9.673410361867373e-05}

 16%|█▌        | 1600/10000 [11:22<30:35,  4.58it/s]
 16%|█▌        | 1601/10000 [11:22<31:03,  4.51it/s]
 16%|█▌        | 1602/10000 [11:22<32:15,  4.34it/s]
 16%|█▌        | 1603/10000 [11:23<34:31,  4.05it/s]
 16%|█▌        | 1604/10000 [11:23<35:20,  3.96it/s]
 16%|█▌        | 1605/10000 [11:23<35:36,  3.93it/s]
 16%|█▌        | 1606/10000 [11:23<36:19,  3.85it/s]
 16%|█▌        | 1607/10000 [11:24<33:31,  4.17it/s]
 16%|█▌        | 1608/10000 [11:24<31:05,  4.50it/s]
 16%|█▌        | 1609/10000 [11:24<29:34,  4.73it/s]
 16%|█▌        | 1610/10000 [11:24<28:32,  4.90it/s]
                                                    
{'loss': 0.3883, 'grad_norm': 1.0488473176956177, 'learning_rate': 9.667506985324909e-05}

 16%|█▌        | 1610/10000 [11:24<28:32,  4.90it/s]
 16%|█▌        | 1611/10000 [11:24<28:38,  4.88it/s]
 16%|█▌        | 1612/10000 [11:25<29:20,  4.77it/s]
 16%|█▌        | 1613/10000 [11:25<30:00,  4.66it/s]
 16%|█▌        | 1614/10000 [11:25<29:56,  4.67it/s]
 16%|█▌        | 1615/10000 [11:25<30:48,  4.54it/s]
 16%|█▌        | 1616/10000 [11:25<30:00,  4.66it/s]
 16%|█▌        | 1617/10000 [11:26<29:57,  4.66it/s]
 16%|█▌        | 1618/10000 [11:26<28:48,  4.85it/s]
 16%|█▌        | 1619/10000 [11:26<28:30,  4.90it/s]
 16%|█▌        | 1620/10000 [11:26<27:34,  5.06it/s]
                                                    
{'loss': 0.3605, 'grad_norm': 0.7647603154182434, 'learning_rate': 9.661552565673855e-05}

 16%|█▌        | 1620/10000 [11:26<27:34,  5.06it/s]
 16%|█▌        | 1621/10000 [11:26<28:28,  4.90it/s]
 16%|█▌        | 1622/10000 [11:27<29:47,  4.69it/s]
 16%|█▌        | 1623/10000 [11:27<30:04,  4.64it/s]
 16%|█▌        | 1624/10000 [11:27<30:11,  4.62it/s]
 16%|█▋        | 1625/10000 [11:27<30:26,  4.59it/s]
 16%|█▋        | 1626/10000 [11:28<29:04,  4.80it/s]
 16%|█▋        | 1627/10000 [11:28<28:07,  4.96it/s]
 16%|█▋        | 1628/10000 [11:28<27:16,  5.11it/s]
 16%|█▋        | 1629/10000 [11:28<27:12,  5.13it/s]
 16%|█▋        | 1630/10000 [11:28<30:23,  4.59it/s]
                                                    
{'loss': 0.3428, 'grad_norm': 0.9906419515609741, 'learning_rate': 9.655547168030789e-05}

 16%|█▋        | 1630/10000 [11:28<30:23,  4.59it/s]
 16%|█▋        | 1631/10000 [11:29<32:04,  4.35it/s]
 16%|█▋        | 1632/10000 [11:29<32:43,  4.26it/s]
 16%|█▋        | 1633/10000 [11:29<32:57,  4.23it/s]
 16%|█▋        | 1634/10000 [11:29<31:33,  4.42it/s]Rank 0, Worker 2: Wait for shard 9 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 16%|█▋        | 1635/10000 [11:30<30:12,  4.62it/s]
 16%|█▋        | 1636/10000 [11:30<29:17,  4.76it/s]
 16%|█▋        | 1637/10000 [11:30<29:12,  4.77it/s]
 16%|█▋        | 1638/10000 [11:30<30:11,  4.62it/s]
 16%|█▋        | 1639/10000 [11:30<31:36,  4.41it/s]
 16%|█▋        | 1640/10000 [11:31<32:20,  4.31it/s]
                                                    
{'loss': 0.3647, 'grad_norm': 0.8446850180625916, 'learning_rate': 9.649490858069777e-05}

 16%|█▋        | 1640/10000 [11:31<32:20,  4.31it/s]
 16%|█▋        | 1641/10000 [11:31<33:31,  4.16it/s]
 16%|█▋        | 1642/10000 [11:31<32:16,  4.32it/s]
 16%|█▋        | 1643/10000 [11:31<30:29,  4.57it/s]
 16%|█▋        | 1644/10000 [11:32<28:51,  4.83it/s]
 16%|█▋        | 1645/10000 [11:32<28:36,  4.87it/s]
 16%|█▋        | 1646/10000 [11:32<29:17,  4.75it/s]
 16%|█▋        | 1647/10000 [11:32<30:12,  4.61it/s]
 16%|█▋        | 1648/10000 [11:32<30:57,  4.50it/s]
 16%|█▋        | 1649/10000 [11:33<31:21,  4.44it/s]
 16%|█▋        | 1650/10000 [11:33<31:09,  4.47it/s]
                                                    
{'loss': 0.3334, 'grad_norm': 1.0439382791519165, 'learning_rate': 9.643383702021658e-05}

 16%|█▋        | 1650/10000 [11:33<31:09,  4.47it/s]
 17%|█▋        | 1651/10000 [11:33<31:11,  4.46it/s]
 17%|█▋        | 1652/10000 [11:33<30:03,  4.63it/s]
 17%|█▋        | 1653/10000 [11:33<29:55,  4.65it/s]
 17%|█▋        | 1654/10000 [11:34<30:15,  4.60it/s]
 17%|█▋        | 1655/10000 [11:34<31:22,  4.43it/s]
 17%|█▋        | 1656/10000 [11:34<32:27,  4.28it/s]
 17%|█▋        | 1657/10000 [11:34<33:15,  4.18it/s]
 17%|█▋        | 1658/10000 [11:35<32:44,  4.25it/s]
 17%|█▋        | 1659/10000 [11:35<31:41,  4.39it/s]
 17%|█▋        | 1660/10000 [11:35<29:55,  4.65it/s]
                                                    
{'loss': 0.3786, 'grad_norm': 0.9220156669616699, 'learning_rate': 9.637225766673307e-05}

 17%|█▋        | 1660/10000 [11:35<29:55,  4.65it/s]
 17%|█▋        | 1661/10000 [11:35<29:43,  4.68it/s]
 17%|█▋        | 1662/10000 [11:36<29:46,  4.67it/s]
 17%|█▋        | 1663/10000 [11:36<29:30,  4.71it/s]
 17%|█▋        | 1664/10000 [11:36<29:11,  4.76it/s]
 17%|█▋        | 1665/10000 [11:36<29:48,  4.66it/s]
 17%|█▋        | 1666/10000 [11:36<29:29,  4.71it/s]
 17%|█▋        | 1667/10000 [11:37<28:56,  4.80it/s]
 17%|█▋        | 1668/10000 [11:37<28:31,  4.87it/s]
 17%|█▋        | 1669/10000 [11:37<28:14,  4.92it/s]
 17%|█▋        | 1670/10000 [11:37<28:04,  4.95it/s]
                                                    
{'loss': 0.3324, 'grad_norm': 0.9172224998474121, 'learning_rate': 9.631017119366922e-05}

 17%|█▋        | 1670/10000 [11:37<28:04,  4.95it/s]
 17%|█▋        | 1671/10000 [11:37<28:27,  4.88it/s]
 17%|█▋        | 1672/10000 [11:38<28:43,  4.83it/s]
 17%|█▋        | 1673/10000 [11:38<29:05,  4.77it/s]
 17%|█▋        | 1674/10000 [11:38<29:05,  4.77it/s]
 17%|█▋        | 1675/10000 [11:38<28:58,  4.79it/s]
 17%|█▋        | 1676/10000 [11:38<29:10,  4.75it/s]
 17%|█▋        | 1677/10000 [11:39<29:06,  4.76it/s]
 17%|█▋        | 1678/10000 [11:39<28:46,  4.82it/s]
 17%|█▋        | 1679/10000 [11:39<28:20,  4.89it/s]
 17%|█▋        | 1680/10000 [11:39<29:40,  4.67it/s]
                                                    
{'loss': 0.3499, 'grad_norm': 1.1384994983673096, 'learning_rate': 9.624757827999273e-05}

 17%|█▋        | 1680/10000 [11:39<29:40,  4.67it/s]
 17%|█▋        | 1681/10000 [11:39<30:01,  4.62it/s]
 17%|█▋        | 1682/10000 [11:40<30:40,  4.52it/s]
 17%|█▋        | 1683/10000 [11:40<30:59,  4.47it/s]
 17%|█▋        | 1684/10000 [11:40<32:09,  4.31it/s]
 17%|█▋        | 1685/10000 [11:40<31:58,  4.33it/s]
 17%|█▋        | 1686/10000 [11:41<30:51,  4.49it/s]
 17%|█▋        | 1687/10000 [11:41<29:44,  4.66it/s]
 17%|█▋        | 1688/10000 [11:41<28:42,  4.83it/s]
 17%|█▋        | 1689/10000 [11:41<28:28,  4.87it/s]
 17%|█▋        | 1690/10000 [11:41<28:35,  4.85it/s]
                                                    
{'loss': 0.3232, 'grad_norm': 0.9601126909255981, 'learning_rate': 9.618447961020971e-05}

 17%|█▋        | 1690/10000 [11:41<28:35,  4.85it/s]
 17%|█▋        | 1691/10000 [11:42<30:09,  4.59it/s]
 17%|█▋        | 1692/10000 [11:42<31:29,  4.40it/s]
 17%|█▋        | 1693/10000 [11:42<31:50,  4.35it/s]
 17%|█▋        | 1694/10000 [11:42<31:42,  4.37it/s]
 17%|█▋        | 1695/10000 [11:43<31:36,  4.38it/s]
 17%|█▋        | 1696/10000 [11:43<30:57,  4.47it/s]
 17%|█▋        | 1697/10000 [11:43<30:10,  4.59it/s]
 17%|█▋        | 1698/10000 [11:43<31:30,  4.39it/s]
 17%|█▋        | 1699/10000 [11:43<30:30,  4.54it/s]
 17%|█▋        | 1700/10000 [11:44<31:40,  4.37it/s]
                                                    
{'loss': 0.3381, 'grad_norm': 1.112043857574463, 'learning_rate': 9.612087587435707e-05}

 17%|█▋        | 1700/10000 [11:44<31:40,  4.37it/s]
 17%|█▋        | 1701/10000 [11:44<32:54,  4.20it/s]
 17%|█▋        | 1702/10000 [11:44<32:24,  4.27it/s]
 17%|█▋        | 1703/10000 [11:44<34:21,  4.02it/s]
 17%|█▋        | 1704/10000 [11:45<33:14,  4.16it/s]
 17%|█▋        | 1705/10000 [11:45<32:44,  4.22it/s]
 17%|█▋        | 1706/10000 [11:45<31:12,  4.43it/s]
 17%|█▋        | 1707/10000 [11:45<29:27,  4.69it/s]
 17%|█▋        | 1708/10000 [11:46<29:19,  4.71it/s]
 17%|█▋        | 1709/10000 [11:46<30:03,  4.60it/s]
 17%|█▋        | 1710/10000 [11:46<30:28,  4.53it/s]
                                                    
{'loss': 0.3576, 'grad_norm': 1.0856786966323853, 'learning_rate': 9.605676776799508e-05}

 17%|█▋        | 1710/10000 [11:46<30:28,  4.53it/s]
 17%|█▋        | 1711/10000 [11:46<33:47,  4.09it/s]
 17%|█▋        | 1712/10000 [11:47<34:10,  4.04it/s]
 17%|█▋        | 1713/10000 [11:47<33:09,  4.17it/s]
 17%|█▋        | 1714/10000 [11:47<32:04,  4.31it/s]
 17%|█▋        | 1715/10000 [11:47<30:10,  4.58it/s]
 17%|█▋        | 1716/10000 [11:47<29:41,  4.65it/s]
 17%|█▋        | 1717/10000 [11:48<29:54,  4.61it/s]
 17%|█▋        | 1718/10000 [11:48<30:24,  4.54it/s]
 17%|█▋        | 1719/10000 [11:48<32:10,  4.29it/s]
 17%|█▋        | 1720/10000 [11:48<31:55,  4.32it/s]
                                                    
{'loss': 0.3251, 'grad_norm': 1.177982211112976, 'learning_rate': 9.599215599219973e-05}

 17%|█▋        | 1720/10000 [11:48<31:55,  4.32it/s]
 17%|█▋        | 1721/10000 [11:49<31:48,  4.34it/s]
 17%|█▋        | 1722/10000 [11:49<30:28,  4.53it/s]
 17%|█▋        | 1723/10000 [11:49<29:15,  4.71it/s]
 17%|█▋        | 1724/10000 [11:49<28:48,  4.79it/s]
 17%|█▋        | 1725/10000 [11:49<28:32,  4.83it/s]
 17%|█▋        | 1726/10000 [11:50<28:54,  4.77it/s]
 17%|█▋        | 1727/10000 [11:50<29:28,  4.68it/s]
 17%|█▋        | 1728/10000 [11:50<29:15,  4.71it/s]
 17%|█▋        | 1729/10000 [11:50<29:13,  4.72it/s]
 17%|█▋        | 1730/10000 [11:50<28:52,  4.77it/s]
                                                    
{'loss': 0.3118, 'grad_norm': 1.0804262161254883, 'learning_rate': 9.592704125355505e-05}

 17%|█▋        | 1730/10000 [11:50<28:52,  4.77it/s]
 17%|█▋        | 1731/10000 [11:51<28:44,  4.80it/s]
 17%|█▋        | 1732/10000 [11:51<28:18,  4.87it/s]
 17%|█▋        | 1733/10000 [11:51<28:07,  4.90it/s]
 17%|█▋        | 1734/10000 [11:51<28:03,  4.91it/s]
 17%|█▋        | 1735/10000 [11:51<28:07,  4.90it/s]
 17%|█▋        | 1736/10000 [11:52<29:42,  4.64it/s]
 17%|█▋        | 1737/10000 [11:52<29:15,  4.71it/s]
 17%|█▋        | 1738/10000 [11:52<29:49,  4.62it/s]
 17%|█▋        | 1739/10000 [11:52<29:42,  4.64it/s]
 17%|█▋        | 1740/10000 [11:53<29:28,  4.67it/s]
                                                    
{'loss': 0.3224, 'grad_norm': 1.109397530555725, 'learning_rate': 9.586142426414538e-05}

 17%|█▋        | 1740/10000 [11:53<29:28,  4.67it/s]
 17%|█▋        | 1741/10000 [11:53<29:38,  4.64it/s]
 17%|█▋        | 1742/10000 [11:53<29:33,  4.66it/s]
 17%|█▋        | 1743/10000 [11:53<29:47,  4.62it/s]
 17%|█▋        | 1744/10000 [11:53<30:35,  4.50it/s]
 17%|█▋        | 1745/10000 [11:54<30:29,  4.51it/s]
 17%|█▋        | 1746/10000 [11:54<32:01,  4.30it/s]
 17%|█▋        | 1747/10000 [11:54<31:44,  4.33it/s]
 17%|█▋        | 1748/10000 [11:54<30:10,  4.56it/s]
 17%|█▋        | 1749/10000 [11:55<29:18,  4.69it/s]
 18%|█▊        | 1750/10000 [11:55<28:38,  4.80it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...Rank 0, Worker 4: Caching shard...

Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.09 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.23 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.32 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.61 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.81 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 18.11 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 1750: eval/loss = 0.3136 (35s)

                                                    
{'loss': 0.3219, 'grad_norm': 0.8143112659454346, 'learning_rate': 9.57953057415476e-05}

 18%|█▊        | 1750/10000 [12:30<28:38,  4.80it/s]
 18%|█▊        | 1751/10000 [12:30<24:44:43, 10.80s/it]
 18%|█▊        | 1752/10000 [12:30<17:27:13,  7.62s/it]
 18%|█▊        | 1753/10000 [12:31<12:20:11,  5.39s/it]
 18%|█▊        | 1754/10000 [12:31<8:46:19,  3.83s/it] 
 18%|█▊        | 1755/10000 [12:31<6:16:46,  2.74s/it]
 18%|█▊        | 1756/10000 [12:31<4:32:00,  1.98s/it]
 18%|█▊        | 1757/10000 [12:31<3:18:44,  1.45s/it]
 18%|█▊        | 1758/10000 [12:32<2:27:52,  1.08s/it]
 18%|█▊        | 1759/10000 [12:32<1:51:55,  1.23it/s]
 18%|█▊        | 1760/10000 [12:32<1:26:28,  1.59it/s]
                                                      
{'loss': 0.3365, 'grad_norm': 1.075909972190857, 'learning_rate': 9.572868640882328e-05}

 18%|█▊        | 1760/10000 [12:32<1:26:28,  1.59it/s]
 18%|█▊        | 1761/10000 [12:32<1:08:40,  2.00it/s]
 18%|█▊        | 1762/10000 [12:32<55:20,  2.48it/s]  
 18%|█▊        | 1763/10000 [12:33<46:15,  2.97it/s]
 18%|█▊        | 1764/10000 [12:33<40:06,  3.42it/s]
 18%|█▊        | 1765/10000 [12:33<48:36,  2.82it/s]
 18%|█▊        | 1766/10000 [12:33<42:06,  3.26it/s]
 18%|█▊        | 1767/10000 [12:34<37:22,  3.67it/s]
 18%|█▊        | 1768/10000 [12:34<33:36,  4.08it/s]
 18%|█▊        | 1769/10000 [12:34<31:25,  4.37it/s]
 18%|█▊        | 1770/10000 [12:34<29:52,  4.59it/s]
                                                    
{'loss': 0.2947, 'grad_norm': 1.0702075958251953, 'learning_rate': 9.56615669945108e-05}

 18%|█▊        | 1770/10000 [12:34<29:52,  4.59it/s]
 18%|█▊        | 1771/10000 [12:34<29:08,  4.71it/s]
 18%|█▊        | 1772/10000 [12:35<29:11,  4.70it/s]
 18%|█▊        | 1773/10000 [12:35<29:33,  4.64it/s]
 18%|█▊        | 1774/10000 [12:35<29:03,  4.72it/s]
 18%|█▊        | 1775/10000 [12:35<27:49,  4.93it/s]
 18%|█▊        | 1776/10000 [12:35<27:05,  5.06it/s]
 18%|█▊        | 1777/10000 [12:36<26:10,  5.24it/s]
 18%|█▊        | 1778/10000 [12:36<25:52,  5.29it/s]
 18%|█▊        | 1779/10000 [12:36<25:58,  5.27it/s]
 18%|█▊        | 1780/10000 [12:36<26:15,  5.22it/s]
                                                    
{'loss': 0.3039, 'grad_norm': 0.82491534948349, 'learning_rate': 9.55939482326173e-05}

 18%|█▊        | 1780/10000 [12:36<26:15,  5.22it/s]
 18%|█▊        | 1781/10000 [12:36<26:59,  5.08it/s]
 18%|█▊        | 1782/10000 [12:37<26:55,  5.09it/s]
 18%|█▊        | 1783/10000 [12:37<26:49,  5.11it/s]
 18%|█▊        | 1784/10000 [12:37<27:31,  4.98it/s]
 18%|█▊        | 1785/10000 [12:37<28:05,  4.87it/s]
 18%|█▊        | 1786/10000 [12:37<28:06,  4.87it/s]
 18%|█▊        | 1787/10000 [12:38<27:36,  4.96it/s]
 18%|█▊        | 1788/10000 [12:38<27:06,  5.05it/s]
 18%|█▊        | 1789/10000 [12:38<27:04,  5.06it/s]
 18%|█▊        | 1790/10000 [12:38<27:07,  5.04it/s]
                                                    
{'loss': 0.2975, 'grad_norm': 1.2845618724822998, 'learning_rate': 9.552583086261069e-05}

 18%|█▊        | 1790/10000 [12:38<27:07,  5.04it/s]
 18%|█▊        | 1791/10000 [12:38<27:06,  5.05it/s]
 18%|█▊        | 1792/10000 [12:39<26:47,  5.11it/s]
 18%|█▊        | 1793/10000 [12:39<27:00,  5.07it/s]
 18%|█▊        | 1794/10000 [12:39<27:09,  5.04it/s]
 18%|█▊        | 1795/10000 [12:39<27:59,  4.88it/s]
 18%|█▊        | 1796/10000 [12:39<28:48,  4.75it/s]
 18%|█▊        | 1797/10000 [12:40<29:20,  4.66it/s]
 18%|█▊        | 1798/10000 [12:40<29:55,  4.57it/s]
 18%|█▊        | 1799/10000 [12:40<29:07,  4.69it/s]
 18%|█▊        | 1800/10000 [12:40<27:41,  4.93it/s]
                                                    
{'loss': 0.3152, 'grad_norm': 1.335937738418579, 'learning_rate': 9.545721562941168e-05}

 18%|█▊        | 1800/10000 [12:40<27:41,  4.93it/s]
 18%|█▊        | 1801/10000 [12:40<27:32,  4.96it/s]
 18%|█▊        | 1802/10000 [12:41<27:02,  5.05it/s]
 18%|█▊        | 1803/10000 [12:41<27:04,  5.05it/s]
 18%|█▊        | 1804/10000 [12:41<27:32,  4.96it/s]
 18%|█▊        | 1805/10000 [12:41<27:22,  4.99it/s]
 18%|█▊        | 1806/10000 [12:41<27:11,  5.02it/s]
 18%|█▊        | 1807/10000 [12:42<27:11,  5.02it/s]
 18%|█▊        | 1808/10000 [12:42<26:27,  5.16it/s]
 18%|█▊        | 1809/10000 [12:42<26:11,  5.21it/s]
 18%|█▊        | 1810/10000 [12:42<26:29,  5.15it/s]
                                                    
{'loss': 0.2936, 'grad_norm': 0.8990812301635742, 'learning_rate': 9.538810328338543e-05}

 18%|█▊        | 1810/10000 [12:42<26:29,  5.15it/s]
 18%|█▊        | 1811/10000 [12:42<27:01,  5.05it/s]
 18%|█▊        | 1812/10000 [12:43<27:04,  5.04it/s]
 18%|█▊        | 1813/10000 [12:43<27:07,  5.03it/s]
 18%|█▊        | 1814/10000 [12:43<26:52,  5.08it/s]
 18%|█▊        | 1815/10000 [12:43<26:12,  5.20it/s]
 18%|█▊        | 1816/10000 [12:43<25:55,  5.26it/s]
 18%|█▊        | 1817/10000 [12:44<25:57,  5.25it/s]
 18%|█▊        | 1818/10000 [12:44<26:25,  5.16it/s]
 18%|█▊        | 1819/10000 [12:44<27:01,  5.05it/s]
 18%|█▊        | 1820/10000 [12:44<27:19,  4.99it/s]
                                                    
{'loss': 0.2922, 'grad_norm': 1.1800341606140137, 'learning_rate': 9.531849458033349e-05}

 18%|█▊        | 1820/10000 [12:44<27:19,  4.99it/s]
 18%|█▊        | 1821/10000 [12:44<26:54,  5.07it/s]
 18%|█▊        | 1822/10000 [12:45<25:55,  5.26it/s]
 18%|█▊        | 1823/10000 [12:45<25:14,  5.40it/s]
 18%|█▊        | 1824/10000 [12:45<24:51,  5.48it/s]
 18%|█▊        | 1825/10000 [12:45<24:43,  5.51it/s]
 18%|█▊        | 1826/10000 [12:45<25:23,  5.37it/s]
 18%|█▊        | 1827/10000 [12:45<26:48,  5.08it/s]
 18%|█▊        | 1828/10000 [12:46<27:36,  4.93it/s]
 18%|█▊        | 1829/10000 [12:46<28:24,  4.79it/s]
 18%|█▊        | 1830/10000 [12:46<29:29,  4.62it/s]
                                                    
{'loss': 0.3025, 'grad_norm': 1.051785945892334, 'learning_rate': 9.524839028148547e-05}

 18%|█▊        | 1830/10000 [12:46<29:29,  4.62it/s]
 18%|█▊        | 1831/10000 [12:46<31:22,  4.34it/s]
 18%|█▊        | 1832/10000 [12:47<31:49,  4.28it/s]
 18%|█▊        | 1833/10000 [12:47<32:21,  4.21it/s]
 18%|█▊        | 1834/10000 [12:47<33:23,  4.08it/s]
 18%|█▊        | 1835/10000 [12:47<31:54,  4.27it/s]
 18%|█▊        | 1836/10000 [12:48<31:12,  4.36it/s]
 18%|█▊        | 1837/10000 [12:48<30:09,  4.51it/s]
 18%|█▊        | 1838/10000 [12:48<28:37,  4.75it/s]
 18%|█▊        | 1839/10000 [12:48<27:38,  4.92it/s]
 18%|█▊        | 1840/10000 [12:48<28:05,  4.84it/s]
                                                    
{'loss': 0.277, 'grad_norm': 1.2061504125595093, 'learning_rate': 9.517779115349077e-05}

 18%|█▊        | 1840/10000 [12:48<28:05,  4.84it/s]
 18%|█▊        | 1841/10000 [12:49<28:41,  4.74it/s]
 18%|█▊        | 1842/10000 [12:49<28:35,  4.75it/s]
 18%|█▊        | 1843/10000 [12:49<28:29,  4.77it/s]
 18%|█▊        | 1844/10000 [12:49<28:22,  4.79it/s]
 18%|█▊        | 1845/10000 [12:49<27:38,  4.92it/s]
 18%|█▊        | 1846/10000 [12:50<27:04,  5.02it/s]
 18%|█▊        | 1847/10000 [12:50<26:28,  5.13it/s]
 18%|█▊        | 1848/10000 [12:50<25:51,  5.25it/s]
 18%|█▊        | 1849/10000 [12:50<25:29,  5.33it/s]
 18%|█▊        | 1850/10000 [12:50<26:21,  5.15it/s]
                                                    
{'loss': 0.2793, 'grad_norm': 0.9401715397834778, 'learning_rate': 9.510669796841014e-05}

 18%|█▊        | 1850/10000 [12:50<26:21,  5.15it/s]
 19%|█▊        | 1851/10000 [12:51<27:14,  4.98it/s]
 19%|█▊        | 1852/10000 [12:51<27:21,  4.96it/s]
 19%|█▊        | 1853/10000 [12:51<27:19,  4.97it/s]
 19%|█▊        | 1854/10000 [12:51<27:12,  4.99it/s]
 19%|█▊        | 1855/10000 [12:51<26:18,  5.16it/s]
 19%|█▊        | 1856/10000 [12:52<25:39,  5.29it/s]
 19%|█▊        | 1857/10000 [12:52<25:21,  5.35it/s]
 19%|█▊        | 1858/10000 [12:52<25:24,  5.34it/s]
 19%|█▊        | 1859/10000 [12:52<25:55,  5.23it/s]
 19%|█▊        | 1860/10000 [12:52<26:37,  5.10it/s]
                                                    
{'loss': 0.2738, 'grad_norm': 1.1173608303070068, 'learning_rate': 9.503511150370727e-05}

 19%|█▊        | 1860/10000 [12:52<26:37,  5.10it/s]
 19%|█▊        | 1861/10000 [12:53<27:54,  4.86it/s]
 19%|█▊        | 1862/10000 [12:53<28:39,  4.73it/s]
 19%|█▊        | 1863/10000 [12:53<28:48,  4.71it/s]
 19%|█▊        | 1864/10000 [12:53<27:24,  4.95it/s]
 19%|█▊        | 1865/10000 [12:53<26:38,  5.09it/s]
 19%|█▊        | 1866/10000 [12:54<26:31,  5.11it/s]
 19%|█▊        | 1867/10000 [12:54<26:54,  5.04it/s]
 19%|█▊        | 1868/10000 [12:54<28:46,  4.71it/s]
 19%|█▊        | 1869/10000 [12:54<29:53,  4.53it/s]
 19%|█▊        | 1870/10000 [12:54<30:19,  4.47it/s]
                                                    
{'loss': 0.2599, 'grad_norm': 1.2399342060089111, 'learning_rate': 9.496303254224024e-05}

 19%|█▊        | 1870/10000 [12:54<30:19,  4.47it/s]
 19%|█▊        | 1871/10000 [12:55<30:03,  4.51it/s]
 19%|█▊        | 1872/10000 [12:55<28:14,  4.80it/s]
 19%|█▊        | 1873/10000 [12:55<26:55,  5.03it/s]
 19%|█▊        | 1874/10000 [12:55<26:49,  5.05it/s]
 19%|█▉        | 1875/10000 [12:55<27:22,  4.95it/s]
 19%|█▉        | 1876/10000 [12:56<28:52,  4.69it/s]
 19%|█▉        | 1877/10000 [12:56<30:08,  4.49it/s]
 19%|█▉        | 1878/10000 [12:56<29:50,  4.54it/s]
 19%|█▉        | 1879/10000 [12:56<28:36,  4.73it/s]
 19%|█▉        | 1880/10000 [12:56<27:10,  4.98it/s]
                                                    
{'loss': 0.269, 'grad_norm': 0.9564872980117798, 'learning_rate': 9.489046187225306e-05}

 19%|█▉        | 1880/10000 [12:57<27:10,  4.98it/s]
 19%|█▉        | 1881/10000 [12:57<26:30,  5.11it/s]
 19%|█▉        | 1882/10000 [12:57<26:14,  5.16it/s]
 19%|█▉        | 1883/10000 [12:57<27:14,  4.97it/s]
 19%|█▉        | 1884/10000 [12:57<29:01,  4.66it/s]
 19%|█▉        | 1885/10000 [12:58<29:44,  4.55it/s]
 19%|█▉        | 1886/10000 [12:58<30:43,  4.40it/s]
 19%|█▉        | 1887/10000 [12:58<30:01,  4.50it/s]
 19%|█▉        | 1888/10000 [12:58<28:04,  4.82it/s]
 19%|█▉        | 1889/10000 [12:58<26:50,  5.04it/s]
 19%|█▉        | 1890/10000 [12:59<26:52,  5.03it/s]
                                                    
{'loss': 0.2661, 'grad_norm': 1.0643640756607056, 'learning_rate': 9.481740028736692e-05}

 19%|█▉        | 1890/10000 [12:59<26:52,  5.03it/s]
 19%|█▉        | 1891/10000 [12:59<28:45,  4.70it/s]
 19%|█▉        | 1892/10000 [12:59<29:15,  4.62it/s]
 19%|█▉        | 1893/10000 [12:59<29:47,  4.53it/s]
 19%|█▉        | 1894/10000 [12:59<29:28,  4.58it/s]
 19%|█▉        | 1895/10000 [13:00<28:13,  4.79it/s]
 19%|█▉        | 1896/10000 [13:00<27:06,  4.98it/s]
 19%|█▉        | 1897/10000 [13:00<26:25,  5.11it/s]
 19%|█▉        | 1898/10000 [13:00<26:46,  5.04it/s]
 19%|█▉        | 1899/10000 [13:00<27:41,  4.87it/s]
 19%|█▉        | 1900/10000 [13:01<28:28,  4.74it/s]
                                                    
{'loss': 0.2823, 'grad_norm': 0.8369889855384827, 'learning_rate': 9.474384858657164e-05}

 19%|█▉        | 1900/10000 [13:01<28:28,  4.74it/s]
 19%|█▉        | 1901/10000 [13:01<29:03,  4.64it/s]
 19%|█▉        | 1902/10000 [13:01<28:47,  4.69it/s]
 19%|█▉        | 1903/10000 [13:01<28:11,  4.79it/s]
 19%|█▉        | 1904/10000 [13:01<26:55,  5.01it/s]
 19%|█▉        | 1905/10000 [13:02<26:10,  5.15it/s]
 19%|█▉        | 1906/10000 [13:02<26:14,  5.14it/s]
 19%|█▉        | 1907/10000 [13:02<27:16,  4.94it/s]
 19%|█▉        | 1908/10000 [13:02<28:47,  4.69it/s]
 19%|█▉        | 1909/10000 [13:03<30:06,  4.48it/s]
 19%|█▉        | 1910/10000 [13:03<30:26,  4.43it/s]
                                                    
{'loss': 0.2491, 'grad_norm': 1.0115993022918701, 'learning_rate': 9.466980757421679e-05}

 19%|█▉        | 1910/10000 [13:03<30:26,  4.43it/s]
 19%|█▉        | 1911/10000 [13:03<31:20,  4.30it/s]
 19%|█▉        | 1912/10000 [13:03<31:05,  4.34it/s]
 19%|█▉        | 1913/10000 [13:03<30:11,  4.46it/s]
 19%|█▉        | 1914/10000 [13:04<29:37,  4.55it/s]
 19%|█▉        | 1915/10000 [13:04<29:08,  4.62it/s]
 19%|█▉        | 1916/10000 [13:04<28:36,  4.71it/s]
 19%|█▉        | 1917/10000 [13:04<28:40,  4.70it/s]
 19%|█▉        | 1918/10000 [13:05<28:55,  4.66it/s]
 19%|█▉        | 1919/10000 [13:05<29:13,  4.61it/s]
 19%|█▉        | 1920/10000 [13:05<30:14,  4.45it/s]
                                                    
{'loss': 0.2503, 'grad_norm': 0.8186662197113037, 'learning_rate': 9.459527806000305e-05}

 19%|█▉        | 1920/10000 [13:05<30:14,  4.45it/s]
 19%|█▉        | 1921/10000 [13:05<30:59,  4.35it/s]
 19%|█▉        | 1922/10000 [13:05<30:29,  4.42it/s]
 19%|█▉        | 1923/10000 [13:06<29:53,  4.50it/s]
 19%|█▉        | 1924/10000 [13:06<29:08,  4.62it/s]
 19%|█▉        | 1925/10000 [13:06<28:19,  4.75it/s]
 19%|█▉        | 1926/10000 [13:06<28:23,  4.74it/s]
 19%|█▉        | 1927/10000 [13:07<28:56,  4.65it/s]
 19%|█▉        | 1928/10000 [13:07<29:55,  4.50it/s]
 19%|█▉        | 1929/10000 [13:07<30:07,  4.47it/s]
 19%|█▉        | 1930/10000 [13:07<31:20,  4.29it/s]
                                                    
{'loss': 0.2481, 'grad_norm': 1.0530577898025513, 'learning_rate': 9.452026085897325e-05}

 19%|█▉        | 1930/10000 [13:07<31:20,  4.29it/s]
 19%|█▉        | 1931/10000 [13:07<30:47,  4.37it/s]
 19%|█▉        | 1932/10000 [13:08<30:05,  4.47it/s]
 19%|█▉        | 1933/10000 [13:08<28:51,  4.66it/s]
 19%|█▉        | 1934/10000 [13:08<28:27,  4.72it/s]
 19%|█▉        | 1935/10000 [13:08<28:17,  4.75it/s]
 19%|█▉        | 1936/10000 [13:09<28:36,  4.70it/s]
 19%|█▉        | 1937/10000 [13:09<29:47,  4.51it/s]
 19%|█▉        | 1938/10000 [13:09<30:50,  4.36it/s]
 19%|█▉        | 1939/10000 [13:09<31:40,  4.24it/s]
 19%|█▉        | 1940/10000 [13:09<30:59,  4.33it/s]
                                                    
{'loss': 0.2858, 'grad_norm': 0.9877114295959473, 'learning_rate': 9.444475679150348e-05}

 19%|█▉        | 1940/10000 [13:10<30:59,  4.33it/s]
 19%|█▉        | 1941/10000 [13:10<29:33,  4.55it/s]
 19%|█▉        | 1942/10000 [13:10<28:11,  4.76it/s]
 19%|█▉        | 1943/10000 [13:10<27:27,  4.89it/s]
 19%|█▉        | 1944/10000 [13:10<27:28,  4.89it/s]
 19%|█▉        | 1945/10000 [13:10<27:34,  4.87it/s]
 19%|█▉        | 1946/10000 [13:11<27:49,  4.82it/s]
 19%|█▉        | 1947/10000 [13:11<27:55,  4.80it/s]
 19%|█▉        | 1948/10000 [13:11<27:24,  4.90it/s]
 19%|█▉        | 1949/10000 [13:11<26:58,  4.97it/s]
 20%|█▉        | 1950/10000 [13:11<26:13,  5.12it/s]
                                                    
{'loss': 0.2358, 'grad_norm': 1.2159459590911865, 'learning_rate': 9.436876668329411e-05}

 20%|█▉        | 1950/10000 [13:11<26:13,  5.12it/s]
 20%|█▉        | 1951/10000 [13:12<26:51,  5.00it/s]
 20%|█▉        | 1952/10000 [13:12<27:10,  4.94it/s]
 20%|█▉        | 1953/10000 [13:12<27:24,  4.89it/s]
 20%|█▉        | 1954/10000 [13:12<27:47,  4.83it/s]
 20%|█▉        | 1955/10000 [13:12<27:49,  4.82it/s]
 20%|█▉        | 1956/10000 [13:13<28:00,  4.79it/s]
 20%|█▉        | 1957/10000 [13:13<27:32,  4.87it/s]
 20%|█▉        | 1958/10000 [13:13<27:40,  4.84it/s]
 20%|█▉        | 1959/10000 [13:13<26:49,  5.00it/s]
 20%|█▉        | 1960/10000 [13:13<26:46,  5.00it/s]
                                                    
{'loss': 0.2496, 'grad_norm': 1.2078099250793457, 'learning_rate': 9.429229136536079e-05}

 20%|█▉        | 1960/10000 [13:14<26:46,  5.00it/s]
 20%|█▉        | 1961/10000 [13:14<27:10,  4.93it/s]
 20%|█▉        | 1962/10000 [13:14<27:39,  4.84it/s]
 20%|█▉        | 1963/10000 [13:14<28:26,  4.71it/s]
 20%|█▉        | 1964/10000 [13:14<27:47,  4.82it/s]
 20%|█▉        | 1965/10000 [13:15<26:47,  5.00it/s]
 20%|█▉        | 1966/10000 [13:15<25:50,  5.18it/s]
 20%|█▉        | 1967/10000 [13:15<25:33,  5.24it/s]
 20%|█▉        | 1968/10000 [13:15<25:35,  5.23it/s]
 20%|█▉        | 1969/10000 [13:15<26:27,  5.06it/s]
 20%|█▉        | 1970/10000 [13:16<27:02,  4.95it/s]
                                                    
{'loss': 0.2355, 'grad_norm': 1.0169525146484375, 'learning_rate': 9.421533167402534e-05}

 20%|█▉        | 1970/10000 [13:16<27:02,  4.95it/s]
 20%|█▉        | 1971/10000 [13:16<27:45,  4.82it/s]
 20%|█▉        | 1972/10000 [13:16<27:33,  4.86it/s]
 20%|█▉        | 1973/10000 [13:16<26:32,  5.04it/s]
 20%|█▉        | 1974/10000 [13:16<25:46,  5.19it/s]
 20%|█▉        | 1975/10000 [13:16<25:03,  5.34it/s]
 20%|█▉        | 1976/10000 [13:17<25:05,  5.33it/s]
 20%|█▉        | 1977/10000 [13:17<25:26,  5.26it/s]
 20%|█▉        | 1978/10000 [13:17<25:38,  5.21it/s]
 20%|█▉        | 1979/10000 [13:17<25:58,  5.15it/s]
 20%|█▉        | 1980/10000 [13:17<25:50,  5.17it/s]
                                                    
{'loss': 0.2747, 'grad_norm': 0.8852677345275879, 'learning_rate': 9.413788845090666e-05}

 20%|█▉        | 1980/10000 [13:17<25:50,  5.17it/s]
 20%|█▉        | 1981/10000 [13:18<25:45,  5.19it/s]
 20%|█▉        | 1982/10000 [13:18<25:04,  5.33it/s]
 20%|█▉        | 1983/10000 [13:18<24:49,  5.38it/s]
 20%|█▉        | 1984/10000 [13:18<24:37,  5.43it/s]
 20%|█▉        | 1985/10000 [13:18<24:29,  5.45it/s]
 20%|█▉        | 1986/10000 [13:19<26:22,  5.06it/s]
 20%|█▉        | 1987/10000 [13:19<28:27,  4.69it/s]
 20%|█▉        | 1988/10000 [13:19<29:20,  4.55it/s]
 20%|█▉        | 1989/10000 [13:19<30:06,  4.43it/s]
 20%|█▉        | 1990/10000 [13:20<31:01,  4.30it/s]
                                                    
{'loss': 0.2432, 'grad_norm': 0.889837384223938, 'learning_rate': 9.405996254291136e-05}

 20%|█▉        | 1990/10000 [13:20<31:01,  4.30it/s]
 20%|█▉        | 1991/10000 [13:20<30:38,  4.36it/s]
 20%|█▉        | 1992/10000 [13:20<29:37,  4.51it/s]
 20%|█▉        | 1993/10000 [13:20<28:09,  4.74it/s]
 20%|█▉        | 1994/10000 [13:20<27:51,  4.79it/s]
 20%|█▉        | 1995/10000 [13:21<27:24,  4.87it/s]
 20%|█▉        | 1996/10000 [13:21<28:04,  4.75it/s]
 20%|█▉        | 1997/10000 [13:21<29:13,  4.56it/s]
 20%|█▉        | 1998/10000 [13:21<29:20,  4.55it/s]
 20%|█▉        | 1999/10000 [13:21<30:15,  4.41it/s]
 20%|██        | 2000/10000 [13:22<30:19,  4.40it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.72 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.08 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.12 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.37 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.48 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.59 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 2000: eval/loss = 0.2494 (38s)

                                                    
{'loss': 0.2499, 'grad_norm': 1.0552875995635986, 'learning_rate': 9.398155480222474e-05}

 20%|██        | 2000/10000 [13:59<30:19,  4.40it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-2000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-2000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-2000/wandb_config.json

 20%|██        | 2001/10000 [14:26<43:31:56, 19.59s/it]
 20%|██        | 2002/10000 [14:27<30:35:42, 13.77s/it]
 20%|██        | 2003/10000 [14:27<21:32:50,  9.70s/it]
 20%|██        | 2004/10000 [14:27<15:12:54,  6.85s/it]
 20%|██        | 2005/10000 [14:27<10:47:27,  4.86s/it]
 20%|██        | 2006/10000 [14:27<7:41:19,  3.46s/it] 
 20%|██        | 2007/10000 [14:28<5:31:01,  2.48s/it]
 20%|██        | 2008/10000 [14:28<3:59:42,  1.80s/it]
 20%|██        | 2009/10000 [14:28<2:55:36,  1.32s/it]
 20%|██        | 2010/10000 [14:28<2:09:50,  1.03it/s]
                                                      
{'loss': 0.2445, 'grad_norm': 1.0169700384140015, 'learning_rate': 9.390266608630128e-05}

 20%|██        | 2010/10000 [14:28<2:09:50,  1.03it/s]
 20%|██        | 2011/10000 [14:28<1:38:37,  1.35it/s]
 20%|██        | 2012/10000 [14:29<1:16:12,  1.75it/s]
 20%|██        | 2013/10000 [14:29<1:00:21,  2.21it/s]
 20%|██        | 2014/10000 [14:29<49:30,  2.69it/s]  
 20%|██        | 2015/10000 [14:29<42:09,  3.16it/s]
 20%|██        | 2016/10000 [14:29<37:18,  3.57it/s]
 20%|██        | 2017/10000 [14:30<34:20,  3.87it/s]
 20%|██        | 2018/10000 [14:30<31:54,  4.17it/s]
 20%|██        | 2019/10000 [14:30<30:17,  4.39it/s]
 20%|██        | 2020/10000 [14:30<28:38,  4.64it/s]
                                                    
{'loss': 0.2815, 'grad_norm': 1.0074092149734497, 'learning_rate': 9.38232972578553e-05}

 20%|██        | 2020/10000 [14:30<28:38,  4.64it/s]
 20%|██        | 2021/10000 [14:30<27:30,  4.83it/s]
 20%|██        | 2022/10000 [14:31<26:20,  5.05it/s]
 20%|██        | 2023/10000 [14:31<25:40,  5.18it/s]
 20%|██        | 2024/10000 [14:31<25:17,  5.26it/s]
 20%|██        | 2025/10000 [14:31<25:29,  5.21it/s]
 20%|██        | 2026/10000 [14:31<25:21,  5.24it/s]
 20%|██        | 2027/10000 [14:31<25:36,  5.19it/s]
 20%|██        | 2028/10000 [14:32<25:43,  5.17it/s]
 20%|██        | 2029/10000 [14:32<25:37,  5.18it/s]
 20%|██        | 2030/10000 [14:32<25:26,  5.22it/s]
                                                    
{'loss': 0.2451, 'grad_norm': 0.7891824841499329, 'learning_rate': 9.374344918485164e-05}

 20%|██        | 2030/10000 [14:32<25:26,  5.22it/s]
 20%|██        | 2031/10000 [14:32<25:09,  5.28it/s]
 20%|██        | 2032/10000 [14:32<24:38,  5.39it/s]
 20%|██        | 2033/10000 [14:33<24:13,  5.48it/s]
 20%|██        | 2034/10000 [14:33<23:58,  5.54it/s]
 20%|██        | 2035/10000 [14:33<23:53,  5.56it/s]
 20%|██        | 2036/10000 [14:33<24:12,  5.48it/s]
 20%|██        | 2037/10000 [14:33<24:21,  5.45it/s]
 20%|██        | 2038/10000 [14:34<24:29,  5.42it/s]
 20%|██        | 2039/10000 [14:34<24:30,  5.41it/s]
 20%|██        | 2040/10000 [14:34<24:24,  5.44it/s]
                                                    
{'loss': 0.2451, 'grad_norm': 1.070995807647705, 'learning_rate': 9.366312274049602e-05}

 20%|██        | 2040/10000 [14:34<24:24,  5.44it/s]
 20%|██        | 2041/10000 [14:34<26:25,  5.02it/s]
 20%|██        | 2042/10000 [14:34<26:32,  5.00it/s]
 20%|██        | 2043/10000 [14:34<25:39,  5.17it/s]
 20%|██        | 2044/10000 [14:35<25:52,  5.12it/s]
 20%|██        | 2045/10000 [14:35<25:56,  5.11it/s]
 20%|██        | 2046/10000 [14:35<25:03,  5.29it/s]
 20%|██        | 2047/10000 [14:35<24:33,  5.40it/s]
 20%|██        | 2048/10000 [14:35<24:05,  5.50it/s]
 20%|██        | 2049/10000 [14:36<24:13,  5.47it/s]
 20%|██        | 2050/10000 [14:36<24:28,  5.41it/s]
                                                    
{'loss': 0.2402, 'grad_norm': 1.061820149421692, 'learning_rate': 9.358231880322554e-05}

 20%|██        | 2050/10000 [14:36<24:28,  5.41it/s]
 21%|██        | 2051/10000 [14:36<24:52,  5.33it/s]
 21%|██        | 2052/10000 [14:36<24:44,  5.35it/s]
 21%|██        | 2053/10000 [14:36<24:29,  5.41it/s]
 21%|██        | 2054/10000 [14:37<24:03,  5.51it/s]
 21%|██        | 2055/10000 [14:37<23:44,  5.58it/s]
 21%|██        | 2056/10000 [14:37<23:40,  5.59it/s]
 21%|██        | 2057/10000 [14:37<23:43,  5.58it/s]
 21%|██        | 2058/10000 [14:37<23:44,  5.58it/s]
 21%|██        | 2059/10000 [14:37<23:44,  5.57it/s]
 21%|██        | 2060/10000 [14:38<23:51,  5.55it/s]
                                                    
{'loss': 0.2076, 'grad_norm': 0.9319249987602234, 'learning_rate': 9.350103825669916e-05}

 21%|██        | 2060/10000 [14:38<23:51,  5.55it/s]
 21%|██        | 2061/10000 [14:38<23:58,  5.52it/s]
 21%|██        | 2062/10000 [14:38<23:46,  5.56it/s]
 21%|██        | 2063/10000 [14:38<23:36,  5.60it/s]
 21%|██        | 2064/10000 [14:38<23:24,  5.65it/s]
 21%|██        | 2065/10000 [14:38<23:30,  5.63it/s]
 21%|██        | 2066/10000 [14:39<23:32,  5.62it/s]
 21%|██        | 2067/10000 [14:39<23:30,  5.62it/s]
 21%|██        | 2068/10000 [14:39<23:44,  5.57it/s]
 21%|██        | 2069/10000 [14:39<23:53,  5.53it/s]
 21%|██        | 2070/10000 [14:39<23:38,  5.59it/s]
                                                    
{'loss': 0.2107, 'grad_norm': 1.0968592166900635, 'learning_rate': 9.341928198978787e-05}

 21%|██        | 2070/10000 [14:39<23:38,  5.59it/s]
 21%|██        | 2071/10000 [14:40<23:52,  5.54it/s]
 21%|██        | 2072/10000 [14:40<23:36,  5.60it/s]
 21%|██        | 2073/10000 [14:40<23:20,  5.66it/s]
 21%|██        | 2074/10000 [14:40<23:31,  5.62it/s]
 21%|██        | 2075/10000 [14:40<23:25,  5.64it/s]
 21%|██        | 2076/10000 [14:40<23:20,  5.66it/s]
 21%|██        | 2077/10000 [14:41<23:21,  5.65it/s]
 21%|██        | 2078/10000 [14:41<23:23,  5.64it/s]
 21%|██        | 2079/10000 [14:41<23:19,  5.66it/s]
 21%|██        | 2080/10000 [14:41<23:16,  5.67it/s]
                                                    
{'loss': 0.2404, 'grad_norm': 1.1328935623168945, 'learning_rate': 9.333705089656512e-05}

 21%|██        | 2080/10000 [14:41<23:16,  5.67it/s]
 21%|██        | 2081/10000 [14:41<23:44,  5.56it/s]
 21%|██        | 2082/10000 [14:42<23:35,  5.60it/s]
 21%|██        | 2083/10000 [14:42<23:52,  5.53it/s]
 21%|██        | 2084/10000 [14:42<24:09,  5.46it/s]
 21%|██        | 2085/10000 [14:42<23:55,  5.52it/s]
 21%|██        | 2086/10000 [14:42<23:47,  5.54it/s]
 21%|██        | 2087/10000 [14:42<23:37,  5.58it/s]
 21%|██        | 2088/10000 [14:43<23:30,  5.61it/s]
 21%|██        | 2089/10000 [14:43<23:44,  5.55it/s]
 21%|██        | 2090/10000 [14:43<23:50,  5.53it/s]
                                                    
{'loss': 0.2182, 'grad_norm': 0.9278451800346375, 'learning_rate': 9.325434587629698e-05}

 21%|██        | 2090/10000 [14:43<23:50,  5.53it/s]
 21%|██        | 2091/10000 [14:43<24:00,  5.49it/s]
 21%|██        | 2092/10000 [14:43<23:45,  5.55it/s]
 21%|██        | 2093/10000 [14:44<24:13,  5.44it/s]
 21%|██        | 2094/10000 [14:44<25:47,  5.11it/s]
 21%|██        | 2095/10000 [14:44<27:24,  4.81it/s]
 21%|██        | 2096/10000 [14:44<28:22,  4.64it/s]
 21%|██        | 2097/10000 [14:44<29:01,  4.54it/s]
 21%|██        | 2098/10000 [14:45<29:35,  4.45it/s]
 21%|██        | 2099/10000 [14:45<28:11,  4.67it/s]
 21%|██        | 2100/10000 [14:45<27:17,  4.83it/s]
                                                    
{'loss': 0.2073, 'grad_norm': 1.0966594219207764, 'learning_rate': 9.31711678334323e-05}

 21%|██        | 2100/10000 [14:45<27:17,  4.83it/s]
 21%|██        | 2101/10000 [14:45<27:18,  4.82it/s]
 21%|██        | 2102/10000 [14:45<26:51,  4.90it/s]
 21%|██        | 2103/10000 [14:46<26:53,  4.89it/s]
 21%|██        | 2104/10000 [14:46<28:14,  4.66it/s]
 21%|██        | 2105/10000 [14:46<28:29,  4.62it/s]
 21%|██        | 2106/10000 [14:46<29:22,  4.48it/s]
 21%|██        | 2107/10000 [14:47<29:50,  4.41it/s]
 21%|██        | 2108/10000 [14:47<30:09,  4.36it/s]
 21%|██        | 2109/10000 [14:47<30:12,  4.35it/s]
 21%|██        | 2110/10000 [14:47<30:37,  4.29it/s]
                                                    
{'loss': 0.2127, 'grad_norm': 1.1040889024734497, 'learning_rate': 9.308751767759282e-05}

 21%|██        | 2110/10000 [14:47<30:37,  4.29it/s]
 21%|██        | 2111/10000 [14:48<30:40,  4.29it/s]
 21%|██        | 2112/10000 [14:48<31:14,  4.21it/s]
 21%|██        | 2113/10000 [14:48<31:42,  4.15it/s]
 21%|██        | 2114/10000 [14:48<32:33,  4.04it/s]
 21%|██        | 2115/10000 [14:49<33:28,  3.93it/s]
 21%|██        | 2116/10000 [14:49<32:55,  3.99it/s]
 21%|██        | 2117/10000 [14:49<31:55,  4.11it/s]
 21%|██        | 2118/10000 [14:49<30:25,  4.32it/s]
 21%|██        | 2119/10000 [14:49<29:03,  4.52it/s]
 21%|██        | 2120/10000 [14:50<28:18,  4.64it/s]
                                                    
{'loss': 0.2117, 'grad_norm': 1.078241229057312, 'learning_rate': 9.300339632356325e-05}

 21%|██        | 2120/10000 [14:50<28:18,  4.64it/s]
 21%|██        | 2121/10000 [14:50<28:09,  4.66it/s]
 21%|██        | 2122/10000 [14:50<27:48,  4.72it/s]
 21%|██        | 2123/10000 [14:50<28:11,  4.66it/s]
 21%|██        | 2124/10000 [14:50<28:06,  4.67it/s]
 21%|██▏       | 2125/10000 [14:51<29:10,  4.50it/s]
 21%|██▏       | 2126/10000 [14:51<29:18,  4.48it/s]
 21%|██▏       | 2127/10000 [14:51<29:32,  4.44it/s]
 21%|██▏       | 2128/10000 [14:51<28:29,  4.61it/s]
 21%|██▏       | 2129/10000 [14:52<27:40,  4.74it/s]
 21%|██▏       | 2130/10000 [14:52<27:39,  4.74it/s]
                                                    
{'loss': 0.1997, 'grad_norm': 0.823150634765625, 'learning_rate': 9.291880469128124e-05}

 21%|██▏       | 2130/10000 [14:52<27:39,  4.74it/s]
 21%|██▏       | 2131/10000 [14:52<27:50,  4.71it/s]
 21%|██▏       | 2132/10000 [14:52<28:14,  4.64it/s]
 21%|██▏       | 2133/10000 [14:52<29:02,  4.51it/s]
 21%|██▏       | 2134/10000 [14:53<29:52,  4.39it/s]
 21%|██▏       | 2135/10000 [14:53<29:26,  4.45it/s]
 21%|██▏       | 2136/10000 [14:53<29:48,  4.40it/s]
 21%|██▏       | 2137/10000 [14:53<28:26,  4.61it/s]
 21%|██▏       | 2138/10000 [14:54<27:50,  4.71it/s]
 21%|██▏       | 2139/10000 [14:54<27:47,  4.72it/s]
 21%|██▏       | 2140/10000 [14:54<27:50,  4.71it/s]
                                                    
{'loss': 0.2117, 'grad_norm': 1.0198922157287598, 'learning_rate': 9.283374370582732e-05}

 21%|██▏       | 2140/10000 [14:54<27:50,  4.71it/s]
 21%|██▏       | 2141/10000 [14:54<28:10,  4.65it/s]
 21%|██▏       | 2142/10000 [14:54<28:43,  4.56it/s]
 21%|██▏       | 2143/10000 [14:55<28:39,  4.57it/s]
 21%|██▏       | 2144/10000 [14:55<28:09,  4.65it/s]
 21%|██▏       | 2145/10000 [14:55<27:16,  4.80it/s]
 21%|██▏       | 2146/10000 [14:55<26:53,  4.87it/s]
 21%|██▏       | 2147/10000 [14:55<26:40,  4.91it/s]
 21%|██▏       | 2148/10000 [14:56<27:03,  4.84it/s]
 21%|██▏       | 2149/10000 [14:56<27:27,  4.77it/s]
 22%|██▏       | 2150/10000 [14:56<28:44,  4.55it/s]
                                                    
{'loss': 0.2165, 'grad_norm': 1.0626434087753296, 'learning_rate': 9.274821429741482e-05}

 22%|██▏       | 2150/10000 [14:56<28:44,  4.55it/s]
 22%|██▏       | 2151/10000 [14:56<28:19,  4.62it/s]
 22%|██▏       | 2152/10000 [14:57<28:11,  4.64it/s]
 22%|██▏       | 2153/10000 [14:57<27:41,  4.72it/s]
 22%|██▏       | 2154/10000 [14:57<27:18,  4.79it/s]
 22%|██▏       | 2155/10000 [14:57<27:23,  4.77it/s]
 22%|██▏       | 2156/10000 [14:57<27:12,  4.81it/s]
 22%|██▏       | 2157/10000 [14:58<27:23,  4.77it/s]
 22%|██▏       | 2158/10000 [14:58<28:22,  4.61it/s]
 22%|██▏       | 2159/10000 [14:58<28:55,  4.52it/s]
 22%|██▏       | 2160/10000 [14:58<28:55,  4.52it/s]
                                                    
{'loss': 0.2239, 'grad_norm': 1.301125407218933, 'learning_rate': 9.266221740137961e-05}

 22%|██▏       | 2160/10000 [14:59<28:55,  4.52it/s]
 22%|██▏       | 2161/10000 [14:59<35:25,  3.69it/s]
 22%|██▏       | 2162/10000 [14:59<32:22,  4.04it/s]
 22%|██▏       | 2163/10000 [14:59<30:27,  4.29it/s]
 22%|██▏       | 2164/10000 [14:59<29:12,  4.47it/s]
 22%|██▏       | 2165/10000 [14:59<28:54,  4.52it/s]
 22%|██▏       | 2166/10000 [15:00<28:41,  4.55it/s]
 22%|██▏       | 2167/10000 [15:00<28:04,  4.65it/s]
 22%|██▏       | 2168/10000 [15:00<27:39,  4.72it/s]
 22%|██▏       | 2169/10000 [15:00<26:56,  4.85it/s]
 22%|██▏       | 2170/10000 [15:00<25:59,  5.02it/s]
                                                    
{'loss': 0.21, 'grad_norm': 0.8458558917045593, 'learning_rate': 9.257575395817001e-05}

 22%|██▏       | 2170/10000 [15:01<25:59,  5.02it/s]Rank 0, Worker 4: Wait for shard 23 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 22%|██▏       | 2171/10000 [15:01<26:28,  4.93it/s]
 22%|██▏       | 2172/10000 [15:01<26:26,  4.93it/s]
 22%|██▏       | 2173/10000 [15:01<26:22,  4.94it/s]
 22%|██▏       | 2174/10000 [15:01<27:18,  4.78it/s]
 22%|██▏       | 2175/10000 [15:02<27:34,  4.73it/s]
 22%|██▏       | 2176/10000 [15:02<28:34,  4.56it/s]
 22%|██▏       | 2177/10000 [15:02<29:51,  4.37it/s]
 22%|██▏       | 2178/10000 [15:02<29:12,  4.46it/s]
 22%|██▏       | 2179/10000 [15:02<29:27,  4.43it/s]
 22%|██▏       | 2180/10000 [15:03<28:15,  4.61it/s]
                                                    
{'loss': 0.2004, 'grad_norm': 1.014033317565918, 'learning_rate': 9.248882491333637e-05}

 22%|██▏       | 2180/10000 [15:03<28:15,  4.61it/s]
 22%|██▏       | 2181/10000 [15:03<27:58,  4.66it/s]
 22%|██▏       | 2182/10000 [15:03<28:30,  4.57it/s]
 22%|██▏       | 2183/10000 [15:03<27:58,  4.66it/s]
 22%|██▏       | 2184/10000 [15:03<27:37,  4.71it/s]
 22%|██▏       | 2185/10000 [15:04<27:30,  4.74it/s]
 22%|██▏       | 2186/10000 [15:04<27:01,  4.82it/s]
 22%|██▏       | 2187/10000 [15:04<26:49,  4.85it/s]
 22%|██▏       | 2188/10000 [15:04<27:18,  4.77it/s]
 22%|██▏       | 2189/10000 [15:05<26:55,  4.83it/s]
 22%|██▏       | 2190/10000 [15:05<26:38,  4.89it/s]
                                                    
{'loss': 0.1954, 'grad_norm': 0.8399479389190674, 'learning_rate': 9.240143121752076e-05}

 22%|██▏       | 2190/10000 [15:05<26:38,  4.89it/s]
 22%|██▏       | 2191/10000 [15:05<26:41,  4.88it/s]
 22%|██▏       | 2192/10000 [15:05<26:25,  4.92it/s]
 22%|██▏       | 2193/10000 [15:05<26:21,  4.94it/s]
 22%|██▏       | 2194/10000 [15:06<26:16,  4.95it/s]
 22%|██▏       | 2195/10000 [15:06<26:19,  4.94it/s]
 22%|██▏       | 2196/10000 [15:06<26:32,  4.90it/s]
 22%|██▏       | 2197/10000 [15:06<26:53,  4.84it/s]
 22%|██▏       | 2198/10000 [15:06<26:42,  4.87it/s]
 22%|██▏       | 2199/10000 [15:07<26:33,  4.90it/s]
 22%|██▏       | 2200/10000 [15:07<26:31,  4.90it/s]
                                                    
{'loss': 0.2041, 'grad_norm': 0.8896646499633789, 'learning_rate': 9.23135738264467e-05}

 22%|██▏       | 2200/10000 [15:07<26:31,  4.90it/s]
 22%|██▏       | 2201/10000 [15:07<26:44,  4.86it/s]
 22%|██▏       | 2202/10000 [15:07<26:19,  4.94it/s]
 22%|██▏       | 2203/10000 [15:07<26:09,  4.97it/s]
 22%|██▏       | 2204/10000 [15:08<26:14,  4.95it/s]
 22%|██▏       | 2205/10000 [15:08<26:15,  4.95it/s]
 22%|██▏       | 2206/10000 [15:08<26:33,  4.89it/s]
 22%|██▏       | 2207/10000 [15:08<26:57,  4.82it/s]
 22%|██▏       | 2208/10000 [15:08<27:40,  4.69it/s]
 22%|██▏       | 2209/10000 [15:09<27:44,  4.68it/s]
 22%|██▏       | 2210/10000 [15:09<27:25,  4.74it/s]
                                                    
{'loss': 0.2122, 'grad_norm': 0.8813541531562805, 'learning_rate': 9.222525370090849e-05}

 22%|██▏       | 2210/10000 [15:09<27:25,  4.74it/s]
 22%|██▏       | 2211/10000 [15:09<27:35,  4.71it/s]
 22%|██▏       | 2212/10000 [15:09<27:26,  4.73it/s]
 22%|██▏       | 2213/10000 [15:09<27:10,  4.78it/s]
 22%|██▏       | 2214/10000 [15:10<27:13,  4.77it/s]
 22%|██▏       | 2215/10000 [15:10<27:17,  4.75it/s]
 22%|██▏       | 2216/10000 [15:10<27:06,  4.79it/s]
 22%|██▏       | 2217/10000 [15:10<27:26,  4.73it/s]
 22%|██▏       | 2218/10000 [15:11<27:02,  4.80it/s]
 22%|██▏       | 2219/10000 [15:11<26:47,  4.84it/s]
 22%|██▏       | 2220/10000 [15:11<26:38,  4.87it/s]
                                                    
{'loss': 0.1819, 'grad_norm': 1.1454514265060425, 'learning_rate': 9.213647180676088e-05}

 22%|██▏       | 2220/10000 [15:11<26:38,  4.87it/s]
 22%|██▏       | 2221/10000 [15:11<26:55,  4.81it/s]
 22%|██▏       | 2222/10000 [15:11<27:21,  4.74it/s]
 22%|██▏       | 2223/10000 [15:12<27:05,  4.78it/s]
 22%|██▏       | 2224/10000 [15:12<27:56,  4.64it/s]
 22%|██▏       | 2225/10000 [15:12<27:16,  4.75it/s]
 22%|██▏       | 2226/10000 [15:12<26:56,  4.81it/s]
 22%|██▏       | 2227/10000 [15:12<26:47,  4.84it/s]
 22%|██▏       | 2228/10000 [15:13<26:42,  4.85it/s]
 22%|██▏       | 2229/10000 [15:13<26:37,  4.87it/s]
 22%|██▏       | 2230/10000 [15:13<26:32,  4.88it/s]
                                                    
{'loss': 0.1911, 'grad_norm': 0.8740575909614563, 'learning_rate': 9.204722911490846e-05}

 22%|██▏       | 2230/10000 [15:13<26:32,  4.88it/s]
 22%|██▏       | 2231/10000 [15:13<27:33,  4.70it/s]
 22%|██▏       | 2232/10000 [15:13<28:24,  4.56it/s]
 22%|██▏       | 2233/10000 [15:14<29:10,  4.44it/s]
 22%|██▏       | 2234/10000 [15:14<28:37,  4.52it/s]
 22%|██▏       | 2235/10000 [15:14<27:57,  4.63it/s]
 22%|██▏       | 2236/10000 [15:14<27:21,  4.73it/s]
 22%|██▏       | 2237/10000 [15:15<26:58,  4.80it/s]
 22%|██▏       | 2238/10000 [15:15<27:45,  4.66it/s]
 22%|██▏       | 2239/10000 [15:15<28:17,  4.57it/s]
 22%|██▏       | 2240/10000 [15:15<28:51,  4.48it/s]
                                                    
{'loss': 0.2159, 'grad_norm': 1.2543132305145264, 'learning_rate': 9.1957526601295e-05}

 22%|██▏       | 2240/10000 [15:15<28:51,  4.48it/s]
 22%|██▏       | 2241/10000 [15:15<28:56,  4.47it/s]
 22%|██▏       | 2242/10000 [15:16<28:06,  4.60it/s]
 22%|██▏       | 2243/10000 [15:16<27:10,  4.76it/s]
 22%|██▏       | 2244/10000 [15:16<26:38,  4.85it/s]
 22%|██▏       | 2245/10000 [15:16<26:13,  4.93it/s]
 22%|██▏       | 2246/10000 [15:16<26:12,  4.93it/s]
 22%|██▏       | 2247/10000 [15:17<26:21,  4.90it/s]
 22%|██▏       | 2248/10000 [15:17<26:26,  4.89it/s]
 22%|██▏       | 2249/10000 [15:17<26:22,  4.90it/s]
 22%|██▎       | 2250/10000 [15:17<26:31,  4.87it/s]Rank 0, Worker 2: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.73 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.00 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.17 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.29 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.30 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.34 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 2250: eval/loss = 0.2051 (35s)

                                                    
{'loss': 0.1933, 'grad_norm': 0.9959540963172913, 'learning_rate': 9.186736524689281e-05}

 22%|██▎       | 2250/10000 [15:52<26:31,  4.87it/s]
 23%|██▎       | 2251/10000 [15:52<22:58:27, 10.67s/it]
 23%|██▎       | 2252/10000 [15:53<16:12:12,  7.53s/it]
 23%|██▎       | 2253/10000 [15:53<11:27:50,  5.33s/it]
 23%|██▎       | 2254/10000 [15:53<8:08:50,  3.79s/it] 
 23%|██▎       | 2255/10000 [15:53<5:49:32,  2.71s/it]
 23%|██▎       | 2256/10000 [15:53<4:12:00,  1.95s/it]
 23%|██▎       | 2257/10000 [15:54<3:04:06,  1.43s/it]
 23%|██▎       | 2258/10000 [15:54<2:15:59,  1.05s/it]
 23%|██▎       | 2259/10000 [15:54<1:42:37,  1.26it/s]
 23%|██▎       | 2260/10000 [15:54<1:18:44,  1.64it/s]
                                                      
{'loss': 0.175, 'grad_norm': 1.1042733192443848, 'learning_rate': 9.177674603769204e-05}

 23%|██▎       | 2260/10000 [15:54<1:18:44,  1.64it/s]
 23%|██▎       | 2261/10000 [15:54<1:02:41,  2.06it/s]
 23%|██▎       | 2262/10000 [15:54<51:20,  2.51it/s]  
 23%|██▎       | 2263/10000 [15:55<43:27,  2.97it/s]
 23%|██▎       | 2264/10000 [15:55<38:06,  3.38it/s]
 23%|██▎       | 2265/10000 [15:55<34:22,  3.75it/s]
 23%|██▎       | 2266/10000 [15:55<31:32,  4.09it/s]
 23%|██▎       | 2267/10000 [15:55<30:02,  4.29it/s]
 23%|██▎       | 2268/10000 [15:56<28:00,  4.60it/s]
 23%|██▎       | 2269/10000 [15:56<26:37,  4.84it/s]
 23%|██▎       | 2270/10000 [15:56<26:01,  4.95it/s]
                                                    
{'loss': 0.2019, 'grad_norm': 1.072825312614441, 'learning_rate': 9.168566996468983e-05}

 23%|██▎       | 2270/10000 [15:56<26:01,  4.95it/s]
 23%|██▎       | 2271/10000 [15:56<26:05,  4.94it/s]
 23%|██▎       | 2272/10000 [15:56<25:26,  5.06it/s]
 23%|██▎       | 2273/10000 [15:57<25:06,  5.13it/s]
 23%|██▎       | 2274/10000 [15:57<24:28,  5.26it/s]
 23%|██▎       | 2275/10000 [15:57<24:01,  5.36it/s]
 23%|██▎       | 2276/10000 [15:57<23:44,  5.42it/s]
 23%|██▎       | 2277/10000 [15:57<23:53,  5.39it/s]
 23%|██▎       | 2278/10000 [15:57<23:50,  5.40it/s]
 23%|██▎       | 2279/10000 [15:58<24:03,  5.35it/s]
 23%|██▎       | 2280/10000 [15:58<24:06,  5.34it/s]
                                                    
{'loss': 0.1909, 'grad_norm': 1.0621412992477417, 'learning_rate': 9.159413802387951e-05}

 23%|██▎       | 2280/10000 [15:58<24:06,  5.34it/s]
 23%|██▎       | 2281/10000 [15:58<24:36,  5.23it/s]
 23%|██▎       | 2282/10000 [15:58<24:17,  5.30it/s]
 23%|██▎       | 2283/10000 [15:58<23:48,  5.40it/s]
 23%|██▎       | 2284/10000 [15:59<23:21,  5.51it/s]
 23%|██▎       | 2285/10000 [15:59<23:07,  5.56it/s]
 23%|██▎       | 2286/10000 [15:59<22:57,  5.60it/s]
 23%|██▎       | 2287/10000 [15:59<22:48,  5.64it/s]
 23%|██▎       | 2288/10000 [15:59<22:50,  5.63it/s]
 23%|██▎       | 2289/10000 [15:59<22:48,  5.63it/s]
 23%|██▎       | 2290/10000 [16:00<22:54,  5.61it/s]
                                                    
{'loss': 0.1745, 'grad_norm': 0.9369071125984192, 'learning_rate': 9.150215121623974e-05}

 23%|██▎       | 2290/10000 [16:00<22:54,  5.61it/s]
 23%|██▎       | 2291/10000 [16:00<23:53,  5.38it/s]Rank 0, Worker 5: Wait for shard 62 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 23%|██▎       | 2292/10000 [16:00<25:14,  5.09it/s]
 23%|██▎       | 2293/10000 [16:00<25:46,  4.98it/s]
 23%|██▎       | 2294/10000 [16:01<26:12,  4.90it/s]
 23%|██▎       | 2295/10000 [16:01<25:41,  5.00it/s]
 23%|██▎       | 2296/10000 [16:01<24:46,  5.18it/s]
 23%|██▎       | 2297/10000 [16:01<25:10,  5.10it/s]
 23%|██▎       | 2298/10000 [16:01<24:35,  5.22it/s]
 23%|██▎       | 2299/10000 [16:01<24:03,  5.33it/s]
 23%|██▎       | 2300/10000 [16:02<24:12,  5.30it/s]
                                                    
{'loss': 0.1748, 'grad_norm': 1.0990582704544067, 'learning_rate': 9.140971054772349e-05}

 23%|██▎       | 2300/10000 [16:02<24:12,  5.30it/s]
 23%|██▎       | 2301/10000 [16:02<25:08,  5.10it/s]
 23%|██▎       | 2302/10000 [16:02<24:45,  5.18it/s]
 23%|██▎       | 2303/10000 [16:02<24:20,  5.27it/s]
 23%|██▎       | 2304/10000 [16:02<23:52,  5.37it/s]
 23%|██▎       | 2305/10000 [16:03<23:21,  5.49it/s]
 23%|██▎       | 2306/10000 [16:03<23:08,  5.54it/s]
 23%|██▎       | 2307/10000 [16:03<22:57,  5.59it/s]Rank 0, Worker 3: Wait for shard 6 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 23%|██▎       | 2308/10000 [16:03<23:27,  5.47it/s]
 23%|██▎       | 2309/10000 [16:03<23:51,  5.37it/s]
 23%|██▎       | 2310/10000 [16:04<24:13,  5.29it/s]
                                                    
{'loss': 0.1735, 'grad_norm': 1.368462085723877, 'learning_rate': 9.131681702924713e-05}

 23%|██▎       | 2310/10000 [16:04<24:13,  5.29it/s]
 23%|██▎       | 2311/10000 [16:04<24:34,  5.21it/s]
 23%|██▎       | 2312/10000 [16:04<24:34,  5.21it/s]
 23%|██▎       | 2313/10000 [16:04<23:54,  5.36it/s]
 23%|██▎       | 2314/10000 [16:04<23:31,  5.45it/s]
 23%|██▎       | 2315/10000 [16:04<23:32,  5.44it/s]
 23%|██▎       | 2316/10000 [16:05<23:42,  5.40it/s]
 23%|██▎       | 2317/10000 [16:05<24:07,  5.31it/s]Rank 0, Worker 1: Wait for shard 3 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 23%|██▎       | 2318/10000 [16:05<24:54,  5.14it/s]
 23%|██▎       | 2319/10000 [16:05<25:06,  5.10it/s]
 23%|██▎       | 2320/10000 [16:05<25:04,  5.10it/s]
                                                    
{'loss': 0.1704, 'grad_norm': 0.9651881456375122, 'learning_rate': 9.122347167667926e-05}

 23%|██▎       | 2320/10000 [16:05<25:04,  5.10it/s]
 23%|██▎       | 2321/10000 [16:06<26:00,  4.92it/s]
 23%|██▎       | 2322/10000 [16:06<25:49,  4.96it/s]
 23%|██▎       | 2323/10000 [16:06<26:03,  4.91it/s]
 23%|██▎       | 2324/10000 [16:06<26:33,  4.82it/s]
 23%|██▎       | 2325/10000 [16:06<27:12,  4.70it/s]
 23%|██▎       | 2326/10000 [16:07<27:22,  4.67it/s]
 23%|██▎       | 2327/10000 [16:07<27:33,  4.64it/s]
 23%|██▎       | 2328/10000 [16:07<27:49,  4.60it/s]Rank 0, Worker 0: Wait for shard 40 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 23%|██▎       | 2329/10000 [16:07<28:26,  4.50it/s]
 23%|██▎       | 2330/10000 [16:08<28:36,  4.47it/s]
                                                    
{'loss': 0.188, 'grad_norm': 1.1591694355010986, 'learning_rate': 9.112967551082973e-05}

 23%|██▎       | 2330/10000 [16:08<28:36,  4.47it/s]
 23%|██▎       | 2331/10000 [16:08<28:53,  4.42it/s]
 23%|██▎       | 2332/10000 [16:08<28:37,  4.47it/s]
 23%|██▎       | 2333/10000 [16:08<28:08,  4.54it/s]
 23%|██▎       | 2334/10000 [16:08<27:37,  4.62it/s]
 23%|██▎       | 2335/10000 [16:09<26:51,  4.76it/s]
 23%|██▎       | 2336/10000 [16:09<26:57,  4.74it/s]
 23%|██▎       | 2337/10000 [16:09<26:54,  4.75it/s]
 23%|██▎       | 2338/10000 [16:09<26:52,  4.75it/s]
 23%|██▎       | 2339/10000 [16:10<27:00,  4.73it/s]
 23%|██▎       | 2340/10000 [16:10<27:01,  4.72it/s]
                                                    
{'loss': 0.1817, 'grad_norm': 1.324039101600647, 'learning_rate': 9.103542955743835e-05}

 23%|██▎       | 2340/10000 [16:10<27:01,  4.72it/s]
 23%|██▎       | 2341/10000 [16:10<26:23,  4.84it/s]
 23%|██▎       | 2342/10000 [16:10<25:26,  5.02it/s]
 23%|██▎       | 2343/10000 [16:10<24:32,  5.20it/s]
 23%|██▎       | 2344/10000 [16:10<24:45,  5.15it/s]
 23%|██▎       | 2345/10000 [16:11<24:51,  5.13it/s]
 23%|██▎       | 2346/10000 [16:11<25:16,  5.05it/s]
 23%|██▎       | 2347/10000 [16:11<25:35,  4.98it/s]
 23%|██▎       | 2348/10000 [16:11<25:40,  4.97it/s]
 23%|██▎       | 2349/10000 [16:12<26:13,  4.86it/s]
 24%|██▎       | 2350/10000 [16:12<26:19,  4.84it/s]
                                                    
{'loss': 0.1676, 'grad_norm': 1.2204004526138306, 'learning_rate': 9.094073484716381e-05}

 24%|██▎       | 2350/10000 [16:12<26:19,  4.84it/s]
 24%|██▎       | 2351/10000 [16:12<26:02,  4.89it/s]
 24%|██▎       | 2352/10000 [16:12<25:40,  4.96it/s]
 24%|██▎       | 2353/10000 [16:12<25:26,  5.01it/s]
 24%|██▎       | 2354/10000 [16:13<25:31,  4.99it/s]
 24%|██▎       | 2355/10000 [16:13<26:14,  4.86it/s]
 24%|██▎       | 2356/10000 [16:13<26:25,  4.82it/s]
 24%|██▎       | 2357/10000 [16:13<26:15,  4.85it/s]
 24%|██▎       | 2358/10000 [16:13<26:29,  4.81it/s]
 24%|██▎       | 2359/10000 [16:14<26:18,  4.84it/s]
 24%|██▎       | 2360/10000 [16:14<26:16,  4.85it/s]
                                                    
{'loss': 0.2055, 'grad_norm': 1.1595203876495361, 'learning_rate': 9.084559241557226e-05}

 24%|██▎       | 2360/10000 [16:14<26:16,  4.85it/s]
 24%|██▎       | 2361/10000 [16:14<26:46,  4.76it/s]
 24%|██▎       | 2362/10000 [16:14<26:55,  4.73it/s]
 24%|██▎       | 2363/10000 [16:14<26:46,  4.75it/s]
 24%|██▎       | 2364/10000 [16:15<27:28,  4.63it/s]
 24%|██▎       | 2365/10000 [16:15<27:53,  4.56it/s]
 24%|██▎       | 2366/10000 [16:15<27:25,  4.64it/s]
 24%|██▎       | 2367/10000 [16:15<27:07,  4.69it/s]
 24%|██▎       | 2368/10000 [16:15<26:46,  4.75it/s]
 24%|██▎       | 2369/10000 [16:16<26:37,  4.78it/s]
 24%|██▎       | 2370/10000 [16:16<26:18,  4.83it/s]
                                                    
{'loss': 0.1841, 'grad_norm': 0.8688874244689941, 'learning_rate': 9.075000330312608e-05}

 24%|██▎       | 2370/10000 [16:16<26:18,  4.83it/s]
 24%|██▎       | 2371/10000 [16:16<26:32,  4.79it/s]
 24%|██▎       | 2372/10000 [16:16<26:26,  4.81it/s]
 24%|██▎       | 2373/10000 [16:17<26:21,  4.82it/s]
 24%|██▎       | 2374/10000 [16:17<26:24,  4.81it/s]
 24%|██▍       | 2375/10000 [16:17<26:22,  4.82it/s]
 24%|██▍       | 2376/10000 [16:17<26:24,  4.81it/s]
 24%|██▍       | 2377/10000 [16:17<26:07,  4.86it/s]
 24%|██▍       | 2378/10000 [16:18<25:55,  4.90it/s]
 24%|██▍       | 2379/10000 [16:18<25:48,  4.92it/s]
 24%|██▍       | 2380/10000 [16:18<25:47,  4.93it/s]
                                                    
{'loss': 0.1625, 'grad_norm': 1.3413269519805908, 'learning_rate': 9.065396855517253e-05}

 24%|██▍       | 2380/10000 [16:18<25:47,  4.93it/s]
 24%|██▍       | 2381/10000 [16:18<26:07,  4.86it/s]
 24%|██▍       | 2382/10000 [16:18<26:17,  4.83it/s]
 24%|██▍       | 2383/10000 [16:19<26:07,  4.86it/s]
 24%|██▍       | 2384/10000 [16:19<25:51,  4.91it/s]
 24%|██▍       | 2385/10000 [16:19<25:52,  4.91it/s]
 24%|██▍       | 2386/10000 [16:19<25:47,  4.92it/s]
 24%|██▍       | 2387/10000 [16:19<25:21,  5.00it/s]
 24%|██▍       | 2388/10000 [16:20<25:14,  5.03it/s]
 24%|██▍       | 2389/10000 [16:20<25:09,  5.04it/s]
 24%|██▍       | 2390/10000 [16:20<25:00,  5.07it/s]
                                                    
{'loss': 0.1871, 'grad_norm': 0.9392638206481934, 'learning_rate': 9.055748922193219e-05}

 24%|██▍       | 2390/10000 [16:20<25:00,  5.07it/s]
 24%|██▍       | 2391/10000 [16:20<25:06,  5.05it/s]
 24%|██▍       | 2392/10000 [16:20<24:44,  5.13it/s]
 24%|██▍       | 2393/10000 [16:21<24:41,  5.14it/s]
 24%|██▍       | 2394/10000 [16:21<24:44,  5.12it/s]
 24%|██▍       | 2395/10000 [16:21<25:58,  4.88it/s]
 24%|██▍       | 2396/10000 [16:21<25:47,  4.91it/s]Rank 0, Worker 2: Wait for shard 73 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 24%|██▍       | 2397/10000 [16:21<25:34,  4.95it/s]
 24%|██▍       | 2398/10000 [16:22<25:25,  4.98it/s]
 24%|██▍       | 2399/10000 [16:22<25:13,  5.02it/s]
 24%|██▍       | 2400/10000 [16:22<25:27,  4.97it/s]
                                                    
{'loss': 0.1804, 'grad_norm': 0.8570424318313599, 'learning_rate': 9.046056635848761e-05}

 24%|██▍       | 2400/10000 [16:22<25:27,  4.97it/s]
 24%|██▍       | 2401/10000 [16:22<26:01,  4.87it/s]
 24%|██▍       | 2402/10000 [16:22<25:44,  4.92it/s]
 24%|██▍       | 2403/10000 [16:23<25:31,  4.96it/s]
 24%|██▍       | 2404/10000 [16:23<25:08,  5.04it/s]
 24%|██▍       | 2405/10000 [16:23<24:49,  5.10it/s]
 24%|██▍       | 2406/10000 [16:23<24:35,  5.15it/s]
 24%|██▍       | 2407/10000 [16:23<24:08,  5.24it/s]
 24%|██▍       | 2408/10000 [16:24<24:08,  5.24it/s]
 24%|██▍       | 2409/10000 [16:24<25:00,  5.06it/s]
 24%|██▍       | 2410/10000 [16:24<25:32,  4.95it/s]
                                                    
{'loss': 0.1691, 'grad_norm': 0.9605122208595276, 'learning_rate': 9.036320102477169e-05}

 24%|██▍       | 2410/10000 [16:24<25:32,  4.95it/s]
 24%|██▍       | 2411/10000 [16:24<26:03,  4.85it/s]
 24%|██▍       | 2412/10000 [16:24<25:45,  4.91it/s]
 24%|██▍       | 2413/10000 [16:25<25:12,  5.02it/s]
 24%|██▍       | 2414/10000 [16:25<25:23,  4.98it/s]
 24%|██▍       | 2415/10000 [16:25<25:29,  4.96it/s]
 24%|██▍       | 2416/10000 [16:25<25:37,  4.93it/s]
 24%|██▍       | 2417/10000 [16:25<25:29,  4.96it/s]
 24%|██▍       | 2418/10000 [16:26<25:44,  4.91it/s]
 24%|██▍       | 2419/10000 [16:26<26:00,  4.86it/s]
 24%|██▍       | 2420/10000 [16:26<25:44,  4.91it/s]
                                                    
{'loss': 0.1551, 'grad_norm': 1.0445197820663452, 'learning_rate': 9.02653942855561e-05}

 24%|██▍       | 2420/10000 [16:26<25:44,  4.91it/s]
 24%|██▍       | 2421/10000 [16:26<25:32,  4.94it/s]
 24%|██▍       | 2422/10000 [16:26<27:36,  4.57it/s]
 24%|██▍       | 2423/10000 [16:27<26:08,  4.83it/s]
 24%|██▍       | 2424/10000 [16:27<25:53,  4.88it/s]
 24%|██▍       | 2425/10000 [16:27<25:36,  4.93it/s]
 24%|██▍       | 2426/10000 [16:27<25:20,  4.98it/s]
 24%|██▍       | 2427/10000 [16:27<25:16,  4.99it/s]
 24%|██▍       | 2428/10000 [16:28<25:02,  5.04it/s]
 24%|██▍       | 2429/10000 [16:28<24:38,  5.12it/s]
 24%|██▍       | 2430/10000 [16:28<24:28,  5.15it/s]
                                                    
{'loss': 0.1552, 'grad_norm': 1.0570682287216187, 'learning_rate': 9.016714721043971e-05}

 24%|██▍       | 2430/10000 [16:28<24:28,  5.15it/s]
 24%|██▍       | 2431/10000 [16:28<24:24,  5.17it/s]
 24%|██▍       | 2432/10000 [16:28<23:42,  5.32it/s]
 24%|██▍       | 2433/10000 [16:29<23:26,  5.38it/s]
 24%|██▍       | 2434/10000 [16:29<23:03,  5.47it/s]
 24%|██▍       | 2435/10000 [16:29<23:16,  5.42it/s]
 24%|██▍       | 2436/10000 [16:29<23:26,  5.38it/s]
 24%|██▍       | 2437/10000 [16:29<23:15,  5.42it/s]
 24%|██▍       | 2438/10000 [16:29<23:16,  5.41it/s]
 24%|██▍       | 2439/10000 [16:30<23:09,  5.44it/s]
 24%|██▍       | 2440/10000 [16:30<22:54,  5.50it/s]
                                                    
{'loss': 0.1598, 'grad_norm': 0.9345755577087402, 'learning_rate': 9.006846087383675e-05}

 24%|██▍       | 2440/10000 [16:30<22:54,  5.50it/s]
 24%|██▍       | 2441/10000 [16:30<23:01,  5.47it/s]
 24%|██▍       | 2442/10000 [16:30<23:33,  5.35it/s]
 24%|██▍       | 2443/10000 [16:30<23:15,  5.41it/s]
 24%|██▍       | 2444/10000 [16:31<23:18,  5.40it/s]
 24%|██▍       | 2445/10000 [16:31<23:57,  5.26it/s]
 24%|██▍       | 2446/10000 [16:31<24:38,  5.11it/s]
 24%|██▍       | 2447/10000 [16:31<24:11,  5.20it/s]
 24%|██▍       | 2448/10000 [16:31<23:34,  5.34it/s]
 24%|██▍       | 2449/10000 [16:31<23:01,  5.47it/s]
 24%|██▍       | 2450/10000 [16:32<22:35,  5.57it/s]
                                                    
{'loss': 0.1641, 'grad_norm': 1.0995776653289795, 'learning_rate': 8.996933635496523e-05}

 24%|██▍       | 2450/10000 [16:32<22:35,  5.57it/s]
 25%|██▍       | 2451/10000 [16:32<22:42,  5.54it/s]
 25%|██▍       | 2452/10000 [16:32<22:38,  5.56it/s]
 25%|██▍       | 2453/10000 [16:32<22:31,  5.58it/s]
 25%|██▍       | 2454/10000 [16:32<22:46,  5.52it/s]
 25%|██▍       | 2455/10000 [16:33<23:05,  5.45it/s]
 25%|██▍       | 2456/10000 [16:33<23:12,  5.42it/s]
 25%|██▍       | 2457/10000 [16:33<23:03,  5.45it/s]
 25%|██▍       | 2458/10000 [16:33<24:17,  5.18it/s]
 25%|██▍       | 2459/10000 [16:33<24:15,  5.18it/s]
 25%|██▍       | 2460/10000 [16:34<23:45,  5.29it/s]
                                                    
{'loss': 0.1542, 'grad_norm': 1.0021930932998657, 'learning_rate': 8.986977473783498e-05}

 25%|██▍       | 2460/10000 [16:34<23:45,  5.29it/s]
 25%|██▍       | 2461/10000 [16:34<23:48,  5.28it/s]
 25%|██▍       | 2462/10000 [16:34<23:28,  5.35it/s]
 25%|██▍       | 2463/10000 [16:34<23:17,  5.39it/s]
 25%|██▍       | 2464/10000 [16:34<23:29,  5.35it/s]
 25%|██▍       | 2465/10000 [16:34<23:33,  5.33it/s]
 25%|██▍       | 2466/10000 [16:35<23:47,  5.28it/s]
 25%|██▍       | 2467/10000 [16:35<23:43,  5.29it/s]
 25%|██▍       | 2468/10000 [16:35<23:30,  5.34it/s]
 25%|██▍       | 2469/10000 [16:35<23:17,  5.39it/s]
 25%|██▍       | 2470/10000 [16:35<22:57,  5.46it/s]
                                                    
{'loss': 0.1706, 'grad_norm': 0.9481048583984375, 'learning_rate': 8.97697771112359e-05}

 25%|██▍       | 2470/10000 [16:35<22:57,  5.46it/s]
 25%|██▍       | 2471/10000 [16:36<23:04,  5.44it/s]
 25%|██▍       | 2472/10000 [16:36<22:48,  5.50it/s]
 25%|██▍       | 2473/10000 [16:36<22:31,  5.57it/s]
 25%|██▍       | 2474/10000 [16:36<22:57,  5.47it/s]
 25%|██▍       | 2475/10000 [16:36<23:13,  5.40it/s]
 25%|██▍       | 2476/10000 [16:36<23:22,  5.36it/s]
 25%|██▍       | 2477/10000 [16:37<23:19,  5.37it/s]
 25%|██▍       | 2478/10000 [16:37<23:12,  5.40it/s]
 25%|██▍       | 2479/10000 [16:37<23:02,  5.44it/s]
 25%|██▍       | 2480/10000 [16:37<22:47,  5.50it/s]
                                                    
{'loss': 0.1255, 'grad_norm': 1.0423139333724976, 'learning_rate': 8.966934456872602e-05}

 25%|██▍       | 2480/10000 [16:37<22:47,  5.50it/s]
 25%|██▍       | 2481/10000 [16:37<23:22,  5.36it/s]
 25%|██▍       | 2482/10000 [16:38<24:56,  5.02it/s]
 25%|██▍       | 2483/10000 [16:38<26:02,  4.81it/s]
 25%|██▍       | 2484/10000 [16:38<27:00,  4.64it/s]
 25%|██▍       | 2485/10000 [16:38<27:33,  4.55it/s]
 25%|██▍       | 2486/10000 [16:39<27:52,  4.49it/s]
 25%|██▍       | 2487/10000 [16:39<27:13,  4.60it/s]
 25%|██▍       | 2488/10000 [16:39<26:32,  4.72it/s]
 25%|██▍       | 2489/10000 [16:39<25:49,  4.85it/s]
 25%|██▍       | 2490/10000 [16:39<25:23,  4.93it/s]
                                                    
{'loss': 0.1451, 'grad_norm': 1.2045890092849731, 'learning_rate': 8.95684782086195e-05}

 25%|██▍       | 2490/10000 [16:39<25:23,  4.93it/s]
 25%|██▍       | 2491/10000 [16:40<25:31,  4.90it/s]
 25%|██▍       | 2492/10000 [16:40<26:44,  4.68it/s]
 25%|██▍       | 2493/10000 [16:40<26:37,  4.70it/s]
 25%|██▍       | 2494/10000 [16:40<26:26,  4.73it/s]
 25%|██▍       | 2495/10000 [16:40<26:50,  4.66it/s]
 25%|██▍       | 2496/10000 [16:41<26:12,  4.77it/s]
 25%|██▍       | 2497/10000 [16:41<25:23,  4.93it/s]
 25%|██▍       | 2498/10000 [16:41<25:14,  4.95it/s]
 25%|██▍       | 2499/10000 [16:41<25:38,  4.88it/s]
 25%|██▌       | 2500/10000 [16:41<25:10,  4.97it/s]Rank 0, Worker 3: Caching shard...Rank 0, Worker 2: Caching shard...

Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...Rank 0, Worker 4: Caching shard...

Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.76 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.98 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.37 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.55 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.58 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.70 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 2500: eval/loss = 0.1502 (38s)

                                                    
{'loss': 0.1425, 'grad_norm': 1.0377858877182007, 'learning_rate': 8.946717913397476e-05}

 25%|██▌       | 2500/10000 [17:20<25:10,  4.97it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-2500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-2500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-2500/wandb_config.json

 25%|██▌       | 2501/10000 [17:47<41:13:31, 19.79s/it]
 25%|██▌       | 2502/10000 [17:47<28:58:23, 13.91s/it]
 25%|██▌       | 2503/10000 [17:47<20:23:44,  9.79s/it]
 25%|██▌       | 2504/10000 [17:47<14:23:30,  6.91s/it]
 25%|██▌       | 2505/10000 [17:48<10:11:20,  4.89s/it]
 25%|██▌       | 2506/10000 [17:48<7:14:56,  3.48s/it] 
 25%|██▌       | 2507/10000 [17:48<5:12:42,  2.50s/it]
 25%|██▌       | 2508/10000 [17:48<3:46:35,  1.81s/it]
 25%|██▌       | 2509/10000 [17:48<2:45:52,  1.33s/it]
 25%|██▌       | 2510/10000 [17:49<2:04:18,  1.00it/s]
                                                      
{'loss': 0.1535, 'grad_norm': 0.9814800024032593, 'learning_rate': 8.93654484525822e-05}

 25%|██▌       | 2510/10000 [17:49<2:04:18,  1.00it/s]
 25%|██▌       | 2511/10000 [17:49<1:35:22,  1.31it/s]
 25%|██▌       | 2512/10000 [17:49<1:13:53,  1.69it/s]
 25%|██▌       | 2513/10000 [17:49<58:52,  2.12it/s]  
 25%|██▌       | 2514/10000 [17:50<48:10,  2.59it/s]
 25%|██▌       | 2515/10000 [17:50<40:33,  3.08it/s]
 25%|██▌       | 2516/10000 [17:50<35:35,  3.51it/s]
 25%|██▌       | 2517/10000 [17:50<32:04,  3.89it/s]
 25%|██▌       | 2518/10000 [17:50<30:00,  4.16it/s]
 25%|██▌       | 2519/10000 [17:50<29:22,  4.25it/s]
 25%|██▌       | 2520/10000 [17:51<28:39,  4.35it/s]
                                                    
{'loss': 0.1536, 'grad_norm': 0.9192479252815247, 'learning_rate': 8.926328727695226e-05}

 25%|██▌       | 2520/10000 [17:51<28:39,  4.35it/s]
 25%|██▌       | 2521/10000 [17:51<28:06,  4.43it/s]
 25%|██▌       | 2522/10000 [17:51<26:57,  4.62it/s]
 25%|██▌       | 2523/10000 [17:51<25:34,  4.87it/s]
 25%|██▌       | 2524/10000 [17:51<24:56,  5.00it/s]
 25%|██▌       | 2525/10000 [17:52<24:10,  5.15it/s]
 25%|██▌       | 2526/10000 [17:52<23:39,  5.27it/s]
 25%|██▌       | 2527/10000 [17:52<23:35,  5.28it/s]
 25%|██▌       | 2528/10000 [17:52<23:32,  5.29it/s]
 25%|██▌       | 2529/10000 [17:52<23:42,  5.25it/s]
 25%|██▌       | 2530/10000 [17:53<24:01,  5.18it/s]
                                                    
{'loss': 0.1452, 'grad_norm': 1.0358673334121704, 'learning_rate': 8.916069672430319e-05}

 25%|██▌       | 2530/10000 [17:53<24:01,  5.18it/s]
 25%|██▌       | 2531/10000 [17:53<24:14,  5.14it/s]
 25%|██▌       | 2532/10000 [17:53<23:54,  5.21it/s]
 25%|██▌       | 2533/10000 [17:53<23:32,  5.28it/s]
 25%|██▌       | 2534/10000 [17:53<22:59,  5.41it/s]
 25%|██▌       | 2535/10000 [17:54<22:43,  5.48it/s]
 25%|██▌       | 2536/10000 [17:54<22:32,  5.52it/s]
 25%|██▌       | 2537/10000 [17:54<22:46,  5.46it/s]
 25%|██▌       | 2538/10000 [17:54<23:06,  5.38it/s]
 25%|██▌       | 2539/10000 [17:54<23:33,  5.28it/s]
 25%|██▌       | 2540/10000 [17:54<23:26,  5.30it/s]
                                                    
{'loss': 0.1713, 'grad_norm': 0.9884759783744812, 'learning_rate': 8.905767791654884e-05}

 25%|██▌       | 2540/10000 [17:55<23:26,  5.30it/s]
 25%|██▌       | 2541/10000 [17:55<23:35,  5.27it/s]
 25%|██▌       | 2542/10000 [17:55<23:08,  5.37it/s]
 25%|██▌       | 2543/10000 [17:55<22:44,  5.47it/s]
 25%|██▌       | 2544/10000 [17:55<22:31,  5.52it/s]
 25%|██▌       | 2545/10000 [17:55<22:22,  5.55it/s]
 25%|██▌       | 2546/10000 [17:56<22:15,  5.58it/s]
 25%|██▌       | 2547/10000 [17:56<22:50,  5.44it/s]
 25%|██▌       | 2548/10000 [17:56<23:07,  5.37it/s]
 25%|██▌       | 2549/10000 [17:56<23:17,  5.33it/s]
 26%|██▌       | 2550/10000 [17:56<23:23,  5.31it/s]
                                                    
{'loss': 0.1312, 'grad_norm': 0.9764975309371948, 'learning_rate': 8.895423198028638e-05}

 26%|██▌       | 2550/10000 [17:56<23:23,  5.31it/s]
 26%|██▌       | 2551/10000 [17:57<23:26,  5.30it/s]
 26%|██▌       | 2552/10000 [17:57<22:54,  5.42it/s]
 26%|██▌       | 2553/10000 [17:57<22:30,  5.51it/s]
 26%|██▌       | 2554/10000 [17:57<22:20,  5.56it/s]
 26%|██▌       | 2555/10000 [17:57<22:13,  5.58it/s]
 26%|██▌       | 2556/10000 [17:57<22:33,  5.50it/s]
 26%|██▌       | 2557/10000 [17:58<22:41,  5.47it/s]
 26%|██▌       | 2558/10000 [17:58<22:52,  5.42it/s]
 26%|██▌       | 2559/10000 [17:58<22:24,  5.53it/s]
 26%|██▌       | 2560/10000 [17:58<22:16,  5.56it/s]
                                                    
{'loss': 0.1267, 'grad_norm': 1.0950679779052734, 'learning_rate': 8.885036004678402e-05}

 26%|██▌       | 2560/10000 [17:58<22:16,  5.56it/s]
 26%|██▌       | 2561/10000 [17:58<22:28,  5.52it/s]
 26%|██▌       | 2562/10000 [17:58<22:09,  5.60it/s]
 26%|██▌       | 2563/10000 [17:59<22:06,  5.61it/s]
 26%|██▌       | 2564/10000 [17:59<22:12,  5.58it/s]
 26%|██▌       | 2565/10000 [17:59<21:56,  5.65it/s]
 26%|██▌       | 2566/10000 [17:59<21:55,  5.65it/s]
 26%|██▌       | 2567/10000 [17:59<21:55,  5.65it/s]
 26%|██▌       | 2568/10000 [18:00<21:49,  5.68it/s]
 26%|██▌       | 2569/10000 [18:00<21:56,  5.65it/s]
 26%|██▌       | 2570/10000 [18:00<21:52,  5.66it/s]
                                                    
{'loss': 0.1538, 'grad_norm': 1.0493266582489014, 'learning_rate': 8.874606325196857e-05}

 26%|██▌       | 2570/10000 [18:00<21:52,  5.66it/s]
 26%|██▌       | 2571/10000 [18:00<21:59,  5.63it/s]
 26%|██▌       | 2572/10000 [18:00<21:53,  5.65it/s]
 26%|██▌       | 2573/10000 [18:00<21:46,  5.68it/s]
 26%|██▌       | 2574/10000 [18:01<21:36,  5.73it/s]
 26%|██▌       | 2575/10000 [18:01<21:34,  5.74it/s]
 26%|██▌       | 2576/10000 [18:01<21:37,  5.72it/s]
 26%|██▌       | 2577/10000 [18:01<21:36,  5.73it/s]
 26%|██▌       | 2578/10000 [18:01<21:29,  5.76it/s]
 26%|██▌       | 2579/10000 [18:01<21:34,  5.73it/s]
 26%|██▌       | 2580/10000 [18:02<21:30,  5.75it/s]
                                                    
{'loss': 0.137, 'grad_norm': 1.2003095149993896, 'learning_rate': 8.864134273641304e-05}

 26%|██▌       | 2580/10000 [18:02<21:30,  5.75it/s]
 26%|██▌       | 2581/10000 [18:02<21:43,  5.69it/s]
 26%|██▌       | 2582/10000 [18:02<21:35,  5.72it/s]
 26%|██▌       | 2583/10000 [18:02<21:33,  5.73it/s]
 26%|██▌       | 2584/10000 [18:02<21:52,  5.65it/s]
 26%|██▌       | 2585/10000 [18:03<22:54,  5.39it/s]
 26%|██▌       | 2586/10000 [18:03<24:13,  5.10it/s]
 26%|██▌       | 2587/10000 [18:03<25:28,  4.85it/s]
 26%|██▌       | 2588/10000 [18:03<26:25,  4.67it/s]
 26%|██▌       | 2589/10000 [18:03<26:19,  4.69it/s]
 26%|██▌       | 2590/10000 [18:04<26:16,  4.70it/s]
                                                    
{'loss': 0.1479, 'grad_norm': 1.1008723974227905, 'learning_rate': 8.853619964532427e-05}

 26%|██▌       | 2590/10000 [18:04<26:16,  4.70it/s]
 26%|██▌       | 2591/10000 [18:04<25:11,  4.90it/s]
 26%|██▌       | 2592/10000 [18:04<24:06,  5.12it/s]
 26%|██▌       | 2593/10000 [18:04<23:37,  5.23it/s]
 26%|██▌       | 2594/10000 [18:04<23:23,  5.28it/s]
 26%|██▌       | 2595/10000 [18:05<24:08,  5.11it/s]
 26%|██▌       | 2596/10000 [18:05<25:24,  4.86it/s]
 26%|██▌       | 2597/10000 [18:05<26:01,  4.74it/s]
 26%|██▌       | 2598/10000 [18:05<25:48,  4.78it/s]
 26%|██▌       | 2599/10000 [18:05<25:36,  4.82it/s]
 26%|██▌       | 2600/10000 [18:06<24:40,  5.00it/s]
                                                    
{'loss': 0.1257, 'grad_norm': 0.8453201651573181, 'learning_rate': 8.843063512853019e-05}

 26%|██▌       | 2600/10000 [18:06<24:40,  5.00it/s]
 26%|██▌       | 2601/10000 [18:06<24:09,  5.10it/s]
 26%|██▌       | 2602/10000 [18:06<23:41,  5.20it/s]
 26%|██▌       | 2603/10000 [18:06<23:22,  5.28it/s]
 26%|██▌       | 2604/10000 [18:06<23:50,  5.17it/s]
 26%|██▌       | 2605/10000 [18:07<24:43,  4.99it/s]
 26%|██▌       | 2606/10000 [18:07<25:50,  4.77it/s]
 26%|██▌       | 2607/10000 [18:07<27:05,  4.55it/s]
 26%|██▌       | 2608/10000 [18:07<28:01,  4.40it/s]
 26%|██▌       | 2609/10000 [18:08<27:58,  4.40it/s]
 26%|██▌       | 2610/10000 [18:08<27:04,  4.55it/s]
                                                    
{'loss': 0.1196, 'grad_norm': 0.9597755670547485, 'learning_rate': 8.832465034046749e-05}

 26%|██▌       | 2610/10000 [18:08<27:04,  4.55it/s]
 26%|██▌       | 2611/10000 [18:08<27:29,  4.48it/s]
 26%|██▌       | 2612/10000 [18:08<26:52,  4.58it/s]
 26%|██▌       | 2613/10000 [18:08<27:00,  4.56it/s]
 26%|██▌       | 2614/10000 [18:09<28:11,  4.37it/s]
 26%|██▌       | 2615/10000 [18:09<29:10,  4.22it/s]
 26%|██▌       | 2616/10000 [18:09<31:11,  3.95it/s]
 26%|██▌       | 2617/10000 [18:09<30:28,  4.04it/s]
 26%|██▌       | 2618/10000 [18:10<29:19,  4.19it/s]
 26%|██▌       | 2619/10000 [18:10<27:56,  4.40it/s]
 26%|██▌       | 2620/10000 [18:10<27:10,  4.53it/s]
                                                    
{'loss': 0.1184, 'grad_norm': 0.9915949106216431, 'learning_rate': 8.821824644016882e-05}

 26%|██▌       | 2620/10000 [18:10<27:10,  4.53it/s]
 26%|██▌       | 2621/10000 [18:10<26:21,  4.66it/s]
 26%|██▌       | 2622/10000 [18:10<25:51,  4.75it/s]
 26%|██▌       | 2623/10000 [18:11<25:59,  4.73it/s]
 26%|██▌       | 2624/10000 [18:11<26:37,  4.62it/s]
 26%|██▋       | 2625/10000 [18:11<27:51,  4.41it/s]
 26%|██▋       | 2626/10000 [18:11<28:13,  4.35it/s]
 26%|██▋       | 2627/10000 [18:12<27:03,  4.54it/s]
 26%|██▋       | 2628/10000 [18:12<26:12,  4.69it/s]
 26%|██▋       | 2629/10000 [18:12<25:08,  4.89it/s]
 26%|██▋       | 2630/10000 [18:12<25:01,  4.91it/s]
                                                    
{'loss': 0.1212, 'grad_norm': 0.9470176100730896, 'learning_rate': 8.811142459125019e-05}

 26%|██▋       | 2630/10000 [18:12<25:01,  4.91it/s]
 26%|██▋       | 2631/10000 [18:12<25:19,  4.85it/s]
 26%|██▋       | 2632/10000 [18:13<25:20,  4.85it/s]
 26%|██▋       | 2633/10000 [18:13<26:37,  4.61it/s]
 26%|██▋       | 2634/10000 [18:13<26:44,  4.59it/s]
 26%|██▋       | 2635/10000 [18:13<26:37,  4.61it/s]
 26%|██▋       | 2636/10000 [18:13<25:57,  4.73it/s]
 26%|██▋       | 2637/10000 [18:14<25:00,  4.91it/s]
 26%|██▋       | 2638/10000 [18:14<24:33,  5.00it/s]
 26%|██▋       | 2639/10000 [18:14<24:40,  4.97it/s]
 26%|██▋       | 2640/10000 [18:14<25:03,  4.90it/s]
                                                    
{'loss': 0.1327, 'grad_norm': 1.1138802766799927, 'learning_rate': 8.800418596189822e-05}

 26%|██▋       | 2640/10000 [18:14<25:03,  4.90it/s]
 26%|██▋       | 2641/10000 [18:15<26:51,  4.57it/s]
 26%|██▋       | 2642/10000 [18:15<29:08,  4.21it/s]
 26%|██▋       | 2643/10000 [18:15<29:06,  4.21it/s]
 26%|██▋       | 2644/10000 [18:15<28:43,  4.27it/s]
 26%|██▋       | 2645/10000 [18:15<28:10,  4.35it/s]
 26%|██▋       | 2646/10000 [18:16<26:50,  4.57it/s]
 26%|██▋       | 2647/10000 [18:16<26:12,  4.68it/s]
 26%|██▋       | 2648/10000 [18:16<25:52,  4.73it/s]
 26%|██▋       | 2649/10000 [18:16<26:06,  4.69it/s]
 26%|██▋       | 2650/10000 [18:17<26:52,  4.56it/s]
                                                    
{'loss': 0.1264, 'grad_norm': 0.9608763456344604, 'learning_rate': 8.789653172485737e-05}

 26%|██▋       | 2650/10000 [18:17<26:52,  4.56it/s]
 27%|██▋       | 2651/10000 [18:17<27:16,  4.49it/s]
 27%|██▋       | 2652/10000 [18:17<26:49,  4.57it/s]
 27%|██▋       | 2653/10000 [18:17<26:18,  4.65it/s]
 27%|██▋       | 2654/10000 [18:17<25:31,  4.80it/s]
 27%|██▋       | 2655/10000 [18:18<25:20,  4.83it/s]
 27%|██▋       | 2656/10000 [18:18<25:14,  4.85it/s]
 27%|██▋       | 2657/10000 [18:18<25:11,  4.86it/s]
 27%|██▋       | 2658/10000 [18:18<25:13,  4.85it/s]
 27%|██▋       | 2659/10000 [18:18<25:22,  4.82it/s]
 27%|██▋       | 2660/10000 [18:19<25:28,  4.80it/s]
                                                    
{'loss': 0.1361, 'grad_norm': 1.4438965320587158, 'learning_rate': 8.778846305741715e-05}

 27%|██▋       | 2660/10000 [18:19<25:28,  4.80it/s]
 27%|██▋       | 2661/10000 [18:19<25:53,  4.72it/s]
 27%|██▋       | 2662/10000 [18:19<25:30,  4.79it/s]
 27%|██▋       | 2663/10000 [18:19<25:09,  4.86it/s]
 27%|██▋       | 2664/10000 [18:19<24:57,  4.90it/s]
 27%|██▋       | 2665/10000 [18:20<25:14,  4.84it/s]
 27%|██▋       | 2666/10000 [18:20<25:26,  4.80it/s]
 27%|██▋       | 2667/10000 [18:20<26:20,  4.64it/s]
 27%|██▋       | 2668/10000 [18:20<27:15,  4.48it/s]
 27%|██▋       | 2669/10000 [18:21<28:16,  4.32it/s]
 27%|██▋       | 2670/10000 [18:21<27:52,  4.38it/s]
                                                    
{'loss': 0.1227, 'grad_norm': 1.0725312232971191, 'learning_rate': 8.767998114139918e-05}

 27%|██▋       | 2670/10000 [18:21<27:52,  4.38it/s]
 27%|██▋       | 2671/10000 [18:21<27:55,  4.38it/s]
 27%|██▋       | 2672/10000 [18:21<27:16,  4.48it/s]
 27%|██▋       | 2673/10000 [18:21<26:24,  4.62it/s]
 27%|██▋       | 2674/10000 [18:22<25:53,  4.71it/s]
 27%|██▋       | 2675/10000 [18:22<25:37,  4.76it/s]
 27%|██▋       | 2676/10000 [18:22<25:27,  4.80it/s]
 27%|██▋       | 2677/10000 [18:22<25:41,  4.75it/s]
 27%|██▋       | 2678/10000 [18:23<26:32,  4.60it/s]
 27%|██▋       | 2679/10000 [18:23<26:34,  4.59it/s]
 27%|██▋       | 2680/10000 [18:23<26:44,  4.56it/s]
                                                    
{'loss': 0.1592, 'grad_norm': 0.9315901398658752, 'learning_rate': 8.757108716314429e-05}

 27%|██▋       | 2680/10000 [18:23<26:44,  4.56it/s]
 27%|██▋       | 2681/10000 [18:23<26:38,  4.58it/s]
 27%|██▋       | 2682/10000 [18:23<25:53,  4.71it/s]
 27%|██▋       | 2683/10000 [18:24<25:07,  4.85it/s]
 27%|██▋       | 2684/10000 [18:24<24:32,  4.97it/s]
 27%|██▋       | 2685/10000 [18:24<25:07,  4.85it/s]
 27%|██▋       | 2686/10000 [18:24<25:33,  4.77it/s]
 27%|██▋       | 2687/10000 [18:24<25:44,  4.73it/s]
 27%|██▋       | 2688/10000 [18:25<26:24,  4.61it/s]
 27%|██▋       | 2689/10000 [18:25<26:31,  4.59it/s]
 27%|██▋       | 2690/10000 [18:25<26:22,  4.62it/s]
                                                    
{'loss': 0.1276, 'grad_norm': 0.9864495992660522, 'learning_rate': 8.746178231349962e-05}

 27%|██▋       | 2690/10000 [18:25<26:22,  4.62it/s]
 27%|██▋       | 2691/10000 [18:25<26:07,  4.66it/s]
 27%|██▋       | 2692/10000 [18:25<25:03,  4.86it/s]
 27%|██▋       | 2693/10000 [18:26<24:15,  5.02it/s]
 27%|██▋       | 2694/10000 [18:26<27:17,  4.46it/s]
 27%|██▋       | 2695/10000 [18:26<26:56,  4.52it/s]
 27%|██▋       | 2696/10000 [18:26<27:27,  4.43it/s]
 27%|██▋       | 2697/10000 [18:27<26:58,  4.51it/s]
 27%|██▋       | 2698/10000 [18:27<26:28,  4.60it/s]
 27%|██▋       | 2699/10000 [18:27<25:49,  4.71it/s]
 27%|██▋       | 2700/10000 [18:27<25:40,  4.74it/s]
                                                    
{'loss': 0.1336, 'grad_norm': 0.9456494450569153, 'learning_rate': 8.735206778780549e-05}

 27%|██▋       | 2700/10000 [18:27<25:40,  4.74it/s]
 27%|██▋       | 2701/10000 [18:27<25:04,  4.85it/s]
 27%|██▋       | 2702/10000 [18:28<25:27,  4.78it/s]
 27%|██▋       | 2703/10000 [18:28<25:47,  4.72it/s]
 27%|██▋       | 2704/10000 [18:28<25:51,  4.70it/s]
 27%|██▋       | 2705/10000 [18:28<27:01,  4.50it/s]
 27%|██▋       | 2706/10000 [18:29<27:20,  4.45it/s]
 27%|██▋       | 2707/10000 [18:29<27:02,  4.49it/s]
 27%|██▋       | 2708/10000 [18:29<26:19,  4.62it/s]
 27%|██▋       | 2709/10000 [18:29<25:30,  4.77it/s]
 27%|██▋       | 2710/10000 [18:29<25:56,  4.68it/s]
                                                    
{'loss': 0.1214, 'grad_norm': 0.9761070609092712, 'learning_rate': 8.724194478588234e-05}

 27%|██▋       | 2710/10000 [18:29<25:56,  4.68it/s]
 27%|██▋       | 2711/10000 [18:30<26:46,  4.54it/s]
 27%|██▋       | 2712/10000 [18:30<26:33,  4.57it/s]
 27%|██▋       | 2713/10000 [18:30<26:44,  4.54it/s]
 27%|██▋       | 2714/10000 [18:30<26:47,  4.53it/s]
 27%|██▋       | 2715/10000 [18:30<26:34,  4.57it/s]
 27%|██▋       | 2716/10000 [18:31<26:50,  4.52it/s]
 27%|██▋       | 2717/10000 [18:31<26:23,  4.60it/s]
 27%|██▋       | 2718/10000 [18:31<27:08,  4.47it/s]
 27%|██▋       | 2719/10000 [18:31<28:35,  4.25it/s]
 27%|██▋       | 2720/10000 [18:32<29:27,  4.12it/s]
                                                    
{'loss': 0.1593, 'grad_norm': 1.310697317123413, 'learning_rate': 8.713141451201772e-05}

 27%|██▋       | 2720/10000 [18:32<29:27,  4.12it/s]
 27%|██▋       | 2721/10000 [18:32<29:37,  4.10it/s]
 27%|██▋       | 2722/10000 [18:32<28:17,  4.29it/s]
 27%|██▋       | 2723/10000 [18:32<27:11,  4.46it/s]
 27%|██▋       | 2724/10000 [18:33<26:26,  4.59it/s]
 27%|██▋       | 2725/10000 [18:33<26:26,  4.59it/s]
 27%|██▋       | 2726/10000 [18:33<27:02,  4.48it/s]
 27%|██▋       | 2727/10000 [18:33<27:07,  4.47it/s]
 27%|██▋       | 2728/10000 [18:33<27:33,  4.40it/s]
 27%|██▋       | 2729/10000 [18:34<27:54,  4.34it/s]
 27%|██▋       | 2730/10000 [18:34<27:01,  4.48it/s]
                                                    
{'loss': 0.1433, 'grad_norm': 0.9047120809555054, 'learning_rate': 8.702047817495295e-05}

 27%|██▋       | 2730/10000 [18:34<27:01,  4.48it/s]
 27%|██▋       | 2731/10000 [18:34<27:15,  4.44it/s]
 27%|██▋       | 2732/10000 [18:34<27:07,  4.47it/s]
 27%|██▋       | 2733/10000 [18:35<26:03,  4.65it/s]
 27%|██▋       | 2734/10000 [18:35<25:01,  4.84it/s]
 27%|██▋       | 2735/10000 [18:35<24:22,  4.97it/s]
 27%|██▋       | 2736/10000 [18:35<25:05,  4.83it/s]
 27%|██▋       | 2737/10000 [18:35<25:47,  4.69it/s]
 27%|██▋       | 2738/10000 [18:36<26:22,  4.59it/s]
 27%|██▋       | 2739/10000 [18:36<26:23,  4.59it/s]
 27%|██▋       | 2740/10000 [18:36<26:23,  4.59it/s]
                                                    
{'loss': 0.1166, 'grad_norm': 0.9631437063217163, 'learning_rate': 8.69091369878701e-05}

 27%|██▋       | 2740/10000 [18:36<26:23,  4.59it/s]
 27%|██▋       | 2741/10000 [18:36<25:22,  4.77it/s]
 27%|██▋       | 2742/10000 [18:36<24:19,  4.97it/s]
 27%|██▋       | 2743/10000 [18:37<23:36,  5.12it/s]
 27%|██▋       | 2744/10000 [18:37<23:07,  5.23it/s]
 27%|██▋       | 2745/10000 [18:37<23:57,  5.05it/s]
 27%|██▋       | 2746/10000 [18:37<24:42,  4.89it/s]
 27%|██▋       | 2747/10000 [18:37<25:20,  4.77it/s]
 27%|██▋       | 2748/10000 [18:38<25:23,  4.76it/s]
 27%|██▋       | 2749/10000 [18:38<25:09,  4.80it/s]
 28%|██▊       | 2750/10000 [18:38<24:17,  4.97it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Caching shard...Rank 0, Worker 2: Caching shard...

Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.08 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.60 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.61 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.77 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.96 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 18.01 seconds
Rank 0, Worker 0: Caching shard...
[held-out-eval] step 2750: eval/loss = 0.1121 (36s)

                                                    
{'loss': 0.132, 'grad_norm': 0.8545982837677002, 'learning_rate': 8.679739216837849e-05}

 28%|██▊       | 2750/10000 [19:14<24:17,  4.97it/s]
 28%|██▊       | 2751/10000 [19:14<22:13:10, 11.03s/it]
 28%|██▊       | 2752/10000 [19:15<15:40:26,  7.79s/it]
 28%|██▊       | 2753/10000 [19:15<11:06:00,  5.51s/it]
 28%|██▊       | 2754/10000 [19:15<7:53:35,  3.92s/it] 
 28%|██▊       | 2755/10000 [19:15<5:37:52,  2.80s/it]
 28%|██▊       | 2756/10000 [19:15<4:03:05,  2.01s/it]
 28%|██▊       | 2757/10000 [19:16<2:57:23,  1.47s/it]
 28%|██▊       | 2758/10000 [19:16<2:11:20,  1.09s/it]
 28%|██▊       | 2759/10000 [19:16<1:40:23,  1.20it/s]
 28%|██▊       | 2760/10000 [19:16<1:18:02,  1.55it/s]
                                                      
{'loss': 0.141, 'grad_norm': 0.8017613291740417, 'learning_rate': 8.66852449385016e-05}

 28%|██▊       | 2760/10000 [19:16<1:18:02,  1.55it/s]
 28%|██▊       | 2761/10000 [19:16<1:02:01,  1.95it/s]
 28%|██▊       | 2762/10000 [19:17<50:34,  2.39it/s]  
 28%|██▊       | 2763/10000 [19:17<41:48,  2.88it/s]
 28%|██▊       | 2764/10000 [19:17<35:58,  3.35it/s]
 28%|██▊       | 2765/10000 [19:17<32:08,  3.75it/s]
 28%|██▊       | 2766/10000 [19:17<30:02,  4.01it/s]
 28%|██▊       | 2767/10000 [19:18<28:42,  4.20it/s]
 28%|██▊       | 2768/10000 [19:18<27:44,  4.34it/s]
 28%|██▊       | 2769/10000 [19:18<26:53,  4.48it/s]
 28%|██▊       | 2770/10000 [19:18<25:49,  4.67it/s]
                                                    
{'loss': 0.1479, 'grad_norm': 0.974586546421051, 'learning_rate': 8.657269652466356e-05}

 28%|██▊       | 2770/10000 [19:18<25:49,  4.67it/s]
 28%|██▊       | 2771/10000 [19:18<24:45,  4.87it/s]
 28%|██▊       | 2772/10000 [19:19<23:50,  5.05it/s]
 28%|██▊       | 2773/10000 [19:19<23:57,  5.03it/s]
 28%|██▊       | 2774/10000 [19:19<24:00,  5.02it/s]
 28%|██▊       | 2775/10000 [19:19<24:39,  4.88it/s]
 28%|██▊       | 2776/10000 [19:19<25:02,  4.81it/s]
 28%|██▊       | 2777/10000 [19:20<24:59,  4.82it/s]
 28%|██▊       | 2778/10000 [19:20<25:57,  4.64it/s]
 28%|██▊       | 2779/10000 [19:20<25:28,  4.72it/s]
 28%|██▊       | 2780/10000 [19:20<26:01,  4.62it/s]
                                                    
{'loss': 0.13, 'grad_norm': 1.026877999305725, 'learning_rate': 8.645974815767577e-05}

 28%|██▊       | 2780/10000 [19:20<26:01,  4.62it/s]
 28%|██▊       | 2781/10000 [19:20<25:11,  4.78it/s]
 28%|██▊       | 2782/10000 [19:21<24:58,  4.82it/s]
 28%|██▊       | 2783/10000 [19:21<24:48,  4.85it/s]
 28%|██▊       | 2784/10000 [19:21<24:38,  4.88it/s]
 28%|██▊       | 2785/10000 [19:21<24:18,  4.95it/s]
 28%|██▊       | 2786/10000 [19:21<23:18,  5.16it/s]
 28%|██▊       | 2787/10000 [19:22<22:48,  5.27it/s]
 28%|██▊       | 2788/10000 [19:22<22:19,  5.38it/s]
 28%|██▊       | 2789/10000 [19:22<21:55,  5.48it/s]
 28%|██▊       | 2790/10000 [19:22<21:52,  5.49it/s]
                                                    
{'loss': 0.1255, 'grad_norm': 0.9218412041664124, 'learning_rate': 8.634640107272351e-05}

 28%|██▊       | 2790/10000 [19:22<21:52,  5.49it/s]
 28%|██▊       | 2791/10000 [19:22<22:01,  5.45it/s]
 28%|██▊       | 2792/10000 [19:22<21:50,  5.50it/s]
 28%|██▊       | 2793/10000 [19:23<21:25,  5.61it/s]
 28%|██▊       | 2794/10000 [19:23<21:07,  5.69it/s]
 28%|██▊       | 2795/10000 [19:23<20:56,  5.73it/s]
 28%|██▊       | 2796/10000 [19:23<21:07,  5.68it/s]
 28%|██▊       | 2797/10000 [19:23<21:04,  5.70it/s]
 28%|██▊       | 2798/10000 [19:24<20:55,  5.74it/s]
 28%|██▊       | 2799/10000 [19:24<20:49,  5.76it/s]
 28%|██▊       | 2800/10000 [19:24<20:56,  5.73it/s]
                                                    
{'loss': 0.1236, 'grad_norm': 0.9164907932281494, 'learning_rate': 8.623265650935234e-05}

 28%|██▊       | 2800/10000 [19:24<20:56,  5.73it/s]
 28%|██▊       | 2801/10000 [19:24<21:07,  5.68it/s]
 28%|██▊       | 2802/10000 [19:24<21:00,  5.71it/s]
 28%|██▊       | 2803/10000 [19:24<21:00,  5.71it/s]
 28%|██▊       | 2804/10000 [19:25<20:49,  5.76it/s]
 28%|██▊       | 2805/10000 [19:25<20:46,  5.77it/s]
 28%|██▊       | 2806/10000 [19:25<20:48,  5.76it/s]
 28%|██▊       | 2807/10000 [19:25<20:46,  5.77it/s]
 28%|██▊       | 2808/10000 [19:25<20:43,  5.78it/s]
 28%|██▊       | 2809/10000 [19:25<20:46,  5.77it/s]
 28%|██▊       | 2810/10000 [19:26<20:50,  5.75it/s]
                                                    
{'loss': 0.1152, 'grad_norm': 0.921433687210083, 'learning_rate': 8.611851571145456e-05}

 28%|██▊       | 2810/10000 [19:26<20:50,  5.75it/s]
 28%|██▊       | 2811/10000 [19:26<21:48,  5.49it/s]
 28%|██▊       | 2812/10000 [19:26<22:27,  5.34it/s]
 28%|██▊       | 2813/10000 [19:26<23:24,  5.12it/s]
 28%|██▊       | 2814/10000 [19:26<23:47,  5.03it/s]
 28%|██▊       | 2815/10000 [19:27<24:36,  4.87it/s]
 28%|██▊       | 2816/10000 [19:27<25:06,  4.77it/s]
 28%|██▊       | 2817/10000 [19:27<25:33,  4.68it/s]
 28%|██▊       | 2818/10000 [19:27<25:20,  4.72it/s]
 28%|██▊       | 2819/10000 [19:27<25:14,  4.74it/s]
 28%|██▊       | 2820/10000 [19:28<24:49,  4.82it/s]
                                                    
{'loss': 0.1304, 'grad_norm': 0.8561160564422607, 'learning_rate': 8.600397992725566e-05}

 28%|██▊       | 2820/10000 [19:28<24:49,  4.82it/s]
 28%|██▊       | 2821/10000 [19:28<25:29,  4.69it/s]
 28%|██▊       | 2822/10000 [19:28<24:58,  4.79it/s]
 28%|██▊       | 2823/10000 [19:28<24:22,  4.91it/s]
 28%|██▊       | 2824/10000 [19:29<24:03,  4.97it/s]
 28%|██▊       | 2825/10000 [19:29<23:43,  5.04it/s]
 28%|██▊       | 2826/10000 [19:29<23:10,  5.16it/s]
 28%|██▊       | 2827/10000 [19:29<22:35,  5.29it/s]
 28%|██▊       | 2828/10000 [19:29<22:04,  5.41it/s]
 28%|██▊       | 2829/10000 [19:29<21:42,  5.51it/s]
 28%|██▊       | 2830/10000 [19:30<21:36,  5.53it/s]
                                                    
{'loss': 0.1608, 'grad_norm': 1.0351293087005615, 'learning_rate': 8.588905040930061e-05}

 28%|██▊       | 2830/10000 [19:30<21:36,  5.53it/s]
 28%|██▊       | 2831/10000 [19:30<21:49,  5.47it/s]
 28%|██▊       | 2832/10000 [19:30<21:59,  5.43it/s]
 28%|██▊       | 2833/10000 [19:30<22:07,  5.40it/s]
 28%|██▊       | 2834/10000 [19:30<22:09,  5.39it/s]
 28%|██▊       | 2835/10000 [19:31<22:11,  5.38it/s]
 28%|██▊       | 2836/10000 [19:31<22:06,  5.40it/s]
 28%|██▊       | 2837/10000 [19:31<21:53,  5.45it/s]
 28%|██▊       | 2838/10000 [19:31<21:42,  5.50it/s]
 28%|██▊       | 2839/10000 [19:31<21:31,  5.54it/s]
 28%|██▊       | 2840/10000 [19:31<21:17,  5.61it/s]
                                                    
{'loss': 0.132, 'grad_norm': 1.076572299003601, 'learning_rate': 8.577372841444022e-05}

 28%|██▊       | 2840/10000 [19:31<21:17,  5.61it/s]
 28%|██▊       | 2841/10000 [19:32<21:41,  5.50it/s]
 28%|██▊       | 2842/10000 [19:32<21:31,  5.54it/s]
 28%|██▊       | 2843/10000 [19:32<21:16,  5.61it/s]
 28%|██▊       | 2844/10000 [19:32<21:12,  5.62it/s]
 28%|██▊       | 2845/10000 [19:32<21:26,  5.56it/s]
 28%|██▊       | 2846/10000 [19:32<21:18,  5.59it/s]
 28%|██▊       | 2847/10000 [19:33<21:07,  5.64it/s]
 28%|██▊       | 2848/10000 [19:33<21:10,  5.63it/s]
 28%|██▊       | 2849/10000 [19:33<21:01,  5.67it/s]
 28%|██▊       | 2850/10000 [19:33<21:15,  5.60it/s]
                                                    
{'loss': 0.1359, 'grad_norm': 1.0728052854537964, 'learning_rate': 8.565801520381736e-05}

 28%|██▊       | 2850/10000 [19:33<21:15,  5.60it/s]
 29%|██▊       | 2851/10000 [19:33<22:04,  5.40it/s]
 29%|██▊       | 2852/10000 [19:34<21:42,  5.49it/s]
 29%|██▊       | 2853/10000 [19:34<21:24,  5.56it/s]
 29%|██▊       | 2854/10000 [19:34<21:42,  5.49it/s]
 29%|██▊       | 2855/10000 [19:34<21:29,  5.54it/s]
 29%|██▊       | 2856/10000 [19:34<21:25,  5.56it/s]
 29%|██▊       | 2857/10000 [19:34<21:20,  5.58it/s]
 29%|██▊       | 2858/10000 [19:35<21:12,  5.61it/s]
 29%|██▊       | 2859/10000 [19:35<21:37,  5.50it/s]
 29%|██▊       | 2860/10000 [19:35<21:56,  5.42it/s]
                                                    
{'loss': 0.1302, 'grad_norm': 0.939300537109375, 'learning_rate': 8.554191204285313e-05}

 29%|██▊       | 2860/10000 [19:35<21:56,  5.42it/s]
 29%|██▊       | 2861/10000 [19:35<22:56,  5.19it/s]
 29%|██▊       | 2862/10000 [19:35<23:36,  5.04it/s]
 29%|██▊       | 2863/10000 [19:36<23:42,  5.02it/s]
 29%|██▊       | 2864/10000 [19:36<23:35,  5.04it/s]
 29%|██▊       | 2865/10000 [19:36<22:43,  5.23it/s]
 29%|██▊       | 2866/10000 [19:36<22:31,  5.28it/s]
 29%|██▊       | 2867/10000 [19:36<22:26,  5.30it/s]
 29%|██▊       | 2868/10000 [19:37<23:30,  5.06it/s]
 29%|██▊       | 2869/10000 [19:37<23:32,  5.05it/s]
 29%|██▊       | 2870/10000 [19:37<23:38,  5.02it/s]
                                                    
{'loss': 0.1109, 'grad_norm': 0.9795148372650146, 'learning_rate': 8.542542020123315e-05}

 29%|██▊       | 2870/10000 [19:37<23:38,  5.02it/s]
 29%|██▊       | 2871/10000 [19:37<23:55,  4.97it/s]
 29%|██▊       | 2872/10000 [19:37<23:51,  4.98it/s]
 29%|██▊       | 2873/10000 [19:38<23:26,  5.07it/s]
 29%|██▊       | 2874/10000 [19:38<22:48,  5.21it/s]
 29%|██▉       | 2875/10000 [19:38<22:26,  5.29it/s]
 29%|██▉       | 2876/10000 [19:38<22:02,  5.39it/s]
 29%|██▉       | 2877/10000 [19:38<21:52,  5.43it/s]
 29%|██▉       | 2878/10000 [19:39<22:09,  5.35it/s]
 29%|██▉       | 2879/10000 [19:39<22:43,  5.22it/s]
 29%|██▉       | 2880/10000 [19:39<23:03,  5.15it/s]
                                                    
{'loss': 0.1254, 'grad_norm': 0.8189222812652588, 'learning_rate': 8.530854095289347e-05}

 29%|██▉       | 2880/10000 [19:39<23:03,  5.15it/s]
 29%|██▉       | 2881/10000 [19:39<23:42,  5.01it/s]
 29%|██▉       | 2882/10000 [19:39<23:50,  4.98it/s]
 29%|██▉       | 2883/10000 [19:40<23:53,  4.96it/s]
 29%|██▉       | 2884/10000 [19:40<24:22,  4.87it/s]
 29%|██▉       | 2885/10000 [19:40<24:10,  4.91it/s]
 29%|██▉       | 2886/10000 [19:40<23:37,  5.02it/s]
 29%|██▉       | 2887/10000 [19:40<23:01,  5.15it/s]
 29%|██▉       | 2888/10000 [19:41<22:52,  5.18it/s]
 29%|██▉       | 2889/10000 [19:41<22:50,  5.19it/s]
 29%|██▉       | 2890/10000 [19:41<22:47,  5.20it/s]
                                                    
{'loss': 0.1363, 'grad_norm': 0.8327005505561829, 'learning_rate': 8.519127557600688e-05}

 29%|██▉       | 2890/10000 [19:41<22:47,  5.20it/s]
 29%|██▉       | 2891/10000 [19:41<22:58,  5.16it/s]
 29%|██▉       | 2892/10000 [19:41<22:39,  5.23it/s]
 29%|██▉       | 2893/10000 [19:41<22:43,  5.21it/s]
 29%|██▉       | 2894/10000 [19:42<22:37,  5.24it/s]
 29%|██▉       | 2895/10000 [19:42<22:43,  5.21it/s]
 29%|██▉       | 2896/10000 [19:42<22:45,  5.20it/s]
 29%|██▉       | 2897/10000 [19:42<22:36,  5.23it/s]
 29%|██▉       | 2898/10000 [19:42<22:40,  5.22it/s]
 29%|██▉       | 2899/10000 [19:43<22:22,  5.29it/s]
 29%|██▉       | 2900/10000 [19:43<21:58,  5.39it/s]
                                                    
{'loss': 0.1232, 'grad_norm': 0.8563022613525391, 'learning_rate': 8.507362535296871e-05}

 29%|██▉       | 2900/10000 [19:43<21:58,  5.39it/s]
 29%|██▉       | 2901/10000 [19:43<27:42,  4.27it/s]
 29%|██▉       | 2902/10000 [19:43<25:38,  4.61it/s]
 29%|██▉       | 2903/10000 [19:44<24:36,  4.81it/s]
 29%|██▉       | 2904/10000 [19:44<23:44,  4.98it/s]
 29%|██▉       | 2905/10000 [19:44<22:51,  5.17it/s]
 29%|██▉       | 2906/10000 [19:44<22:51,  5.17it/s]
 29%|██▉       | 2907/10000 [19:44<23:00,  5.14it/s]
 29%|██▉       | 2908/10000 [19:44<22:40,  5.21it/s]
 29%|██▉       | 2909/10000 [19:45<22:38,  5.22it/s]
 29%|██▉       | 2910/10000 [19:45<22:25,  5.27it/s]
                                                    
{'loss': 0.1252, 'grad_norm': 0.9863463640213013, 'learning_rate': 8.495559157038299e-05}

 29%|██▉       | 2910/10000 [19:45<22:25,  5.27it/s]
 29%|██▉       | 2911/10000 [19:45<22:16,  5.30it/s]
 29%|██▉       | 2912/10000 [19:45<21:59,  5.37it/s]
 29%|██▉       | 2913/10000 [19:45<21:44,  5.43it/s]
 29%|██▉       | 2914/10000 [19:46<21:35,  5.47it/s]
 29%|██▉       | 2915/10000 [19:46<21:53,  5.39it/s]
 29%|██▉       | 2916/10000 [19:46<22:06,  5.34it/s]
 29%|██▉       | 2917/10000 [19:46<22:26,  5.26it/s]
 29%|██▉       | 2918/10000 [19:46<22:38,  5.21it/s]
 29%|██▉       | 2919/10000 [19:47<22:29,  5.25it/s]
 29%|██▉       | 2920/10000 [19:47<22:05,  5.34it/s]
                                                    
{'loss': 0.1242, 'grad_norm': 0.8366366624832153, 'learning_rate': 8.483717551904823e-05}

 29%|██▉       | 2920/10000 [19:47<22:05,  5.34it/s]
 29%|██▉       | 2921/10000 [19:47<22:19,  5.29it/s]
 29%|██▉       | 2922/10000 [19:47<22:06,  5.34it/s]
 29%|██▉       | 2923/10000 [19:47<21:45,  5.42it/s]
 29%|██▉       | 2924/10000 [19:47<21:38,  5.45it/s]
 29%|██▉       | 2925/10000 [19:48<21:35,  5.46it/s]
 29%|██▉       | 2926/10000 [19:48<21:19,  5.53it/s]
 29%|██▉       | 2927/10000 [19:48<21:15,  5.55it/s]
 29%|██▉       | 2928/10000 [19:48<21:37,  5.45it/s]
 29%|██▉       | 2929/10000 [19:48<21:50,  5.39it/s]
 29%|██▉       | 2930/10000 [19:49<22:11,  5.31it/s]
                                                    
{'loss': 0.1281, 'grad_norm': 0.7308292388916016, 'learning_rate': 8.47183784939434e-05}

 29%|██▉       | 2930/10000 [19:49<22:11,  5.31it/s]
 29%|██▉       | 2931/10000 [19:49<22:44,  5.18it/s]
 29%|██▉       | 2932/10000 [19:49<22:30,  5.23it/s]Rank 0, Worker 4: Wait for shard 31 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 29%|██▉       | 2933/10000 [19:49<22:42,  5.19it/s]
 29%|██▉       | 2934/10000 [19:49<22:30,  5.23it/s]
 29%|██▉       | 2935/10000 [19:49<22:08,  5.32it/s]
 29%|██▉       | 2936/10000 [19:50<22:10,  5.31it/s]
 29%|██▉       | 2937/10000 [19:50<21:55,  5.37it/s]
 29%|██▉       | 2938/10000 [19:50<22:00,  5.35it/s]
 29%|██▉       | 2939/10000 [19:50<22:09,  5.31it/s]
 29%|██▉       | 2940/10000 [19:50<22:02,  5.34it/s]
                                                    
{'loss': 0.1122, 'grad_norm': 1.0988391637802124, 'learning_rate': 8.459920179421374e-05}

 29%|██▉       | 2940/10000 [19:50<22:02,  5.34it/s]
 29%|██▉       | 2941/10000 [19:51<22:23,  5.26it/s]
 29%|██▉       | 2942/10000 [19:51<21:57,  5.36it/s]
 29%|██▉       | 2943/10000 [19:51<21:35,  5.45it/s]
 29%|██▉       | 2944/10000 [19:51<21:22,  5.50it/s]
 29%|██▉       | 2945/10000 [19:51<21:28,  5.48it/s]
 29%|██▉       | 2946/10000 [19:52<21:20,  5.51it/s]
 29%|██▉       | 2947/10000 [19:52<21:30,  5.47it/s]
 29%|██▉       | 2948/10000 [19:52<21:37,  5.44it/s]
 29%|██▉       | 2949/10000 [19:52<21:26,  5.48it/s]
 30%|██▉       | 2950/10000 [19:52<21:28,  5.47it/s]
                                                    
{'loss': 0.1186, 'grad_norm': 1.247576355934143, 'learning_rate': 8.447964672315656e-05}

 30%|██▉       | 2950/10000 [19:52<21:28,  5.47it/s]
 30%|██▉       | 2951/10000 [19:52<21:54,  5.36it/s]
 30%|██▉       | 2952/10000 [19:53<21:39,  5.42it/s]
 30%|██▉       | 2953/10000 [19:53<21:46,  5.39it/s]
 30%|██▉       | 2954/10000 [19:53<21:48,  5.38it/s]
 30%|██▉       | 2955/10000 [19:53<22:06,  5.31it/s]
 30%|██▉       | 2956/10000 [19:53<22:04,  5.32it/s]
 30%|██▉       | 2957/10000 [19:54<22:10,  5.29it/s]
 30%|██▉       | 2958/10000 [19:54<21:53,  5.36it/s]
 30%|██▉       | 2959/10000 [19:54<21:37,  5.43it/s]
 30%|██▉       | 2960/10000 [19:54<21:31,  5.45it/s]
                                                    
{'loss': 0.1111, 'grad_norm': 0.8852250576019287, 'learning_rate': 8.435971458820692e-05}

 30%|██▉       | 2960/10000 [19:54<21:31,  5.45it/s]
 30%|██▉       | 2961/10000 [19:54<21:54,  5.36it/s]
 30%|██▉       | 2962/10000 [19:55<21:58,  5.34it/s]
 30%|██▉       | 2963/10000 [19:55<22:12,  5.28it/s]
 30%|██▉       | 2964/10000 [19:55<22:05,  5.31it/s]
 30%|██▉       | 2965/10000 [19:55<22:00,  5.33it/s]
 30%|██▉       | 2966/10000 [19:55<21:38,  5.42it/s]
 30%|██▉       | 2967/10000 [19:55<21:27,  5.46it/s]
 30%|██▉       | 2968/10000 [19:56<21:15,  5.51it/s]
 30%|██▉       | 2969/10000 [19:56<21:15,  5.51it/s]
 30%|██▉       | 2970/10000 [19:56<21:31,  5.44it/s]
                                                    
{'loss': 0.1321, 'grad_norm': 0.8364087343215942, 'learning_rate': 8.423940670092345e-05}

 30%|██▉       | 2970/10000 [19:56<21:31,  5.44it/s]
 30%|██▉       | 2971/10000 [19:56<22:14,  5.27it/s]
 30%|██▉       | 2972/10000 [19:56<22:26,  5.22it/s]
 30%|██▉       | 2973/10000 [19:57<21:54,  5.34it/s]
 30%|██▉       | 2974/10000 [19:57<21:31,  5.44it/s]
 30%|██▉       | 2975/10000 [19:57<21:14,  5.51it/s]
 30%|██▉       | 2976/10000 [19:57<21:34,  5.43it/s]
 30%|██▉       | 2977/10000 [19:57<22:03,  5.31it/s]
 30%|██▉       | 2978/10000 [19:58<22:50,  5.12it/s]
 30%|██▉       | 2979/10000 [19:58<24:33,  4.77it/s]
 30%|██▉       | 2980/10000 [19:58<26:01,  4.50it/s]
                                                    
{'loss': 0.1107, 'grad_norm': 1.136894941329956, 'learning_rate': 8.411872437697394e-05}

 30%|██▉       | 2980/10000 [19:58<26:01,  4.50it/s]
 30%|██▉       | 2981/10000 [19:58<26:45,  4.37it/s]
 30%|██▉       | 2982/10000 [19:58<26:47,  4.37it/s]
 30%|██▉       | 2983/10000 [19:59<25:36,  4.57it/s]
 30%|██▉       | 2984/10000 [19:59<25:20,  4.61it/s]
 30%|██▉       | 2985/10000 [19:59<24:35,  4.75it/s]
 30%|██▉       | 2986/10000 [19:59<24:12,  4.83it/s]
 30%|██▉       | 2987/10000 [19:59<24:24,  4.79it/s]
 30%|██▉       | 2988/10000 [20:00<25:12,  4.64it/s]
 30%|██▉       | 2989/10000 [20:00<25:10,  4.64it/s]
 30%|██▉       | 2990/10000 [20:00<24:53,  4.69it/s]
                                                    
{'loss': 0.1292, 'grad_norm': 1.0125927925109863, 'learning_rate': 8.399766893612096e-05}

 30%|██▉       | 2990/10000 [20:00<24:53,  4.69it/s]
 30%|██▉       | 2991/10000 [20:00<25:08,  4.64it/s]
 30%|██▉       | 2992/10000 [20:01<24:30,  4.76it/s]
 30%|██▉       | 2993/10000 [20:01<24:00,  4.86it/s]
 30%|██▉       | 2994/10000 [20:01<24:15,  4.81it/s]
 30%|██▉       | 2995/10000 [20:01<26:28,  4.41it/s]
 30%|██▉       | 2996/10000 [20:01<25:33,  4.57it/s]
 30%|██▉       | 2997/10000 [20:02<26:08,  4.47it/s]
 30%|██▉       | 2998/10000 [20:02<26:39,  4.38it/s]
 30%|██▉       | 2999/10000 [20:02<26:23,  4.42it/s]
 30%|███       | 3000/10000 [20:02<26:12,  4.45it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Caching shard...Rank 0, Worker 4: Caching shard...

Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.63 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 18.87 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.16 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.26 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.41 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.74 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 3000: eval/loss = 0.1285 (38s)

                                                    
{'loss': 0.1211, 'grad_norm': 0.9666236639022827, 'learning_rate': 8.38762417022074e-05}

 30%|███       | 3000/10000 [20:40<26:12,  4.45it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-3000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-3000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-3000/wandb_config.json

 30%|███       | 3001/10000 [21:08<38:29:59, 19.80s/it]
 30%|███       | 3002/10000 [21:08<27:03:00, 13.92s/it]
 30%|███       | 3003/10000 [21:08<19:02:16,  9.80s/it]
 30%|███       | 3004/10000 [21:08<13:25:56,  6.91s/it]
 30%|███       | 3005/10000 [21:09<9:30:41,  4.90s/it] 
 30%|███       | 3006/10000 [21:09<6:46:19,  3.49s/it]
 30%|███       | 3007/10000 [21:09<4:51:22,  2.50s/it]
 30%|███       | 3008/10000 [21:09<3:30:59,  1.81s/it]
 30%|███       | 3009/10000 [21:09<2:34:41,  1.33s/it]
 30%|███       | 3010/10000 [21:10<1:55:00,  1.01it/s]
                                                      
{'loss': 0.1364, 'grad_norm': 1.2558808326721191, 'learning_rate': 8.375444400314204e-05}

 30%|███       | 3010/10000 [21:10<1:55:00,  1.01it/s]
 30%|███       | 3011/10000 [21:10<1:27:09,  1.34it/s]
 30%|███       | 3012/10000 [21:10<1:07:37,  1.72it/s]
 30%|███       | 3013/10000 [21:10<54:22,  2.14it/s]  
 30%|███       | 3014/10000 [21:10<45:27,  2.56it/s]
 30%|███       | 3015/10000 [21:11<38:52,  2.99it/s]
 30%|███       | 3016/10000 [21:11<34:15,  3.40it/s]
 30%|███       | 3017/10000 [21:11<30:51,  3.77it/s]Rank 0, Worker 5: Wait for shard 46 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 30%|███       | 3018/10000 [21:11<28:30,  4.08it/s]
 30%|███       | 3019/10000 [21:11<26:27,  4.40it/s]
 30%|███       | 3020/10000 [21:12<25:07,  4.63it/s]
                                                    
{'loss': 0.1402, 'grad_norm': 1.1646627187728882, 'learning_rate': 8.3632277170885e-05}

 30%|███       | 3020/10000 [21:12<25:07,  4.63it/s]
 30%|███       | 3021/10000 [21:12<24:44,  4.70it/s]
 30%|███       | 3022/10000 [21:12<24:20,  4.78it/s]
 30%|███       | 3023/10000 [21:12<24:04,  4.83it/s]
 30%|███       | 3024/10000 [21:12<23:37,  4.92it/s]
 30%|███       | 3025/10000 [21:13<23:24,  4.97it/s]Rank 0, Worker 1: Wait for shard 33 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 30%|███       | 3026/10000 [21:13<23:24,  4.96it/s]
 30%|███       | 3027/10000 [21:13<23:49,  4.88it/s]
 30%|███       | 3028/10000 [21:13<23:55,  4.86it/s]
 30%|███       | 3029/10000 [21:13<24:06,  4.82it/s]
 30%|███       | 3030/10000 [21:14<24:44,  4.70it/s]
                                                    
{'loss': 0.1082, 'grad_norm': 1.0914292335510254, 'learning_rate': 8.350974254143318e-05}

 30%|███       | 3030/10000 [21:14<24:44,  4.70it/s]
 30%|███       | 3031/10000 [21:14<24:31,  4.74it/s]
 30%|███       | 3032/10000 [21:14<24:00,  4.84it/s]
 30%|███       | 3033/10000 [21:14<23:48,  4.88it/s]
 30%|███       | 3034/10000 [21:14<23:41,  4.90it/s]
 30%|███       | 3035/10000 [21:15<23:23,  4.96it/s]
 30%|███       | 3036/10000 [21:15<23:05,  5.02it/s]
 30%|███       | 3037/10000 [21:15<22:49,  5.08it/s]
 30%|███       | 3038/10000 [21:15<22:35,  5.14it/s]
 30%|███       | 3039/10000 [21:15<22:32,  5.15it/s]Rank 0, Worker 3: Wait for shard 58 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 30%|███       | 3040/10000 [21:16<22:53,  5.07it/s]
                                                    
{'loss': 0.124, 'grad_norm': 0.9643756747245789, 'learning_rate': 8.338684145480566e-05}

 30%|███       | 3040/10000 [21:16<22:53,  5.07it/s]
 30%|███       | 3041/10000 [21:16<23:58,  4.84it/s]
 30%|███       | 3042/10000 [21:16<24:13,  4.79it/s]
 30%|███       | 3043/10000 [21:16<25:04,  4.62it/s]
 30%|███       | 3044/10000 [21:16<24:55,  4.65it/s]
 30%|███       | 3045/10000 [21:17<24:35,  4.71it/s]
 30%|███       | 3046/10000 [21:17<24:36,  4.71it/s]
 30%|███       | 3047/10000 [21:17<23:24,  4.95it/s]
 30%|███       | 3048/10000 [21:17<22:39,  5.11it/s]
 30%|███       | 3049/10000 [21:17<24:34,  4.71it/s]
 30%|███       | 3050/10000 [21:18<24:03,  4.81it/s]
                                                    
{'loss': 0.1239, 'grad_norm': 0.9664937257766724, 'learning_rate': 8.326357525502904e-05}

 30%|███       | 3050/10000 [21:18<24:03,  4.81it/s]
 31%|███       | 3051/10000 [21:18<24:42,  4.69it/s]
 31%|███       | 3052/10000 [21:18<25:11,  4.60it/s]
 31%|███       | 3053/10000 [21:18<25:01,  4.63it/s]
 31%|███       | 3054/10000 [21:19<24:33,  4.71it/s]
 31%|███       | 3055/10000 [21:19<24:05,  4.81it/s]
 31%|███       | 3056/10000 [21:19<23:16,  4.97it/s]
 31%|███       | 3057/10000 [21:19<22:27,  5.15it/s]
 31%|███       | 3058/10000 [21:19<22:33,  5.13it/s]
 31%|███       | 3059/10000 [21:20<22:39,  5.11it/s]
 31%|███       | 3060/10000 [21:20<23:00,  5.03it/s]
                                                    
{'loss': 0.1396, 'grad_norm': 0.9044674038887024, 'learning_rate': 8.313994529012273e-05}

 31%|███       | 3060/10000 [21:20<23:00,  5.03it/s]
 31%|███       | 3061/10000 [21:20<23:36,  4.90it/s]
 31%|███       | 3062/10000 [21:20<23:37,  4.90it/s]
 31%|███       | 3063/10000 [21:20<23:56,  4.83it/s]
 31%|███       | 3064/10000 [21:21<23:40,  4.88it/s]
 31%|███       | 3065/10000 [21:21<23:22,  4.94it/s]
 31%|███       | 3066/10000 [21:21<23:08,  5.00it/s]
 31%|███       | 3067/10000 [21:21<23:00,  5.02it/s]
 31%|███       | 3068/10000 [21:21<22:58,  5.03it/s]
 31%|███       | 3069/10000 [21:22<23:10,  4.99it/s]
 31%|███       | 3070/10000 [21:22<23:12,  4.98it/s]
                                                    
{'loss': 0.1501, 'grad_norm': 1.1713777780532837, 'learning_rate': 8.301595291208422e-05}

 31%|███       | 3070/10000 [21:22<23:12,  4.98it/s]
 31%|███       | 3071/10000 [21:22<23:32,  4.91it/s]
 31%|███       | 3072/10000 [21:22<23:28,  4.92it/s]
 31%|███       | 3073/10000 [21:22<23:52,  4.84it/s]
 31%|███       | 3074/10000 [21:23<24:01,  4.80it/s]
 31%|███       | 3075/10000 [21:23<24:31,  4.71it/s]
 31%|███       | 3076/10000 [21:23<24:08,  4.78it/s]
 31%|███       | 3077/10000 [21:23<24:30,  4.71it/s]
 31%|███       | 3078/10000 [21:23<24:12,  4.76it/s]
 31%|███       | 3079/10000 [21:24<24:15,  4.75it/s]
 31%|███       | 3080/10000 [21:24<25:18,  4.56it/s]
                                                    
{'loss': 0.1382, 'grad_norm': 1.2287262678146362, 'learning_rate': 8.289159947687427e-05}

 31%|███       | 3080/10000 [21:24<25:18,  4.56it/s]
 31%|███       | 3081/10000 [21:24<25:06,  4.59it/s]
 31%|███       | 3082/10000 [21:24<25:10,  4.58it/s]
 31%|███       | 3083/10000 [21:25<26:17,  4.39it/s]
 31%|███       | 3084/10000 [21:25<26:11,  4.40it/s]
 31%|███       | 3085/10000 [21:25<26:46,  4.30it/s]
 31%|███       | 3086/10000 [21:25<27:29,  4.19it/s]
 31%|███       | 3087/10000 [21:26<28:13,  4.08it/s]
 31%|███       | 3088/10000 [21:26<27:54,  4.13it/s]
 31%|███       | 3089/10000 [21:26<26:57,  4.27it/s]
 31%|███       | 3090/10000 [21:26<26:16,  4.38it/s]
                                                    
{'loss': 0.1288, 'grad_norm': 1.033851981163025, 'learning_rate': 8.276688634440216e-05}

 31%|███       | 3090/10000 [21:26<26:16,  4.38it/s]
 31%|███       | 3091/10000 [21:26<25:46,  4.47it/s]
 31%|███       | 3092/10000 [21:27<24:51,  4.63it/s]
 31%|███       | 3093/10000 [21:27<24:49,  4.64it/s]
 31%|███       | 3094/10000 [21:27<24:41,  4.66it/s]
 31%|███       | 3095/10000 [21:27<24:58,  4.61it/s]
 31%|███       | 3096/10000 [21:28<25:35,  4.50it/s]
 31%|███       | 3097/10000 [21:28<26:31,  4.34it/s]
 31%|███       | 3098/10000 [21:28<26:39,  4.31it/s]
 31%|███       | 3099/10000 [21:28<26:51,  4.28it/s]
 31%|███       | 3100/10000 [21:28<26:33,  4.33it/s]
                                                    
{'loss': 0.1198, 'grad_norm': 0.9128175973892212, 'learning_rate': 8.26418148785107e-05}

 31%|███       | 3100/10000 [21:28<26:33,  4.33it/s]
 31%|███       | 3101/10000 [21:29<26:29,  4.34it/s]
 31%|███       | 3102/10000 [21:29<26:19,  4.37it/s]
 31%|███       | 3103/10000 [21:29<25:53,  4.44it/s]
 31%|███       | 3104/10000 [21:29<26:23,  4.35it/s]Rank 0, Worker 2: Wait for shard 35 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 31%|███       | 3105/10000 [21:30<28:22,  4.05it/s]
 31%|███       | 3106/10000 [21:30<28:55,  3.97it/s]
 31%|███       | 3107/10000 [21:30<28:55,  3.97it/s]
 31%|███       | 3108/10000 [21:30<28:00,  4.10it/s]
 31%|███       | 3109/10000 [21:31<26:54,  4.27it/s]
 31%|███       | 3110/10000 [21:31<26:30,  4.33it/s]
                                                    
{'loss': 0.1201, 'grad_norm': 0.936797559261322, 'learning_rate': 8.251638644696141e-05}

 31%|███       | 3110/10000 [21:31<26:30,  4.33it/s]
 31%|███       | 3111/10000 [21:31<26:18,  4.36it/s]
 31%|███       | 3112/10000 [21:31<25:53,  4.43it/s]
 31%|███       | 3113/10000 [21:31<25:54,  4.43it/s]
 31%|███       | 3114/10000 [21:32<26:07,  4.39it/s]
 31%|███       | 3115/10000 [21:32<26:40,  4.30it/s]
 31%|███       | 3116/10000 [21:32<26:44,  4.29it/s]
 31%|███       | 3117/10000 [21:32<26:06,  4.39it/s]
 31%|███       | 3118/10000 [21:33<24:51,  4.61it/s]
 31%|███       | 3119/10000 [21:33<23:48,  4.82it/s]
 31%|███       | 3120/10000 [21:33<23:39,  4.85it/s]
                                                    
{'loss': 0.1247, 'grad_norm': 1.0767229795455933, 'learning_rate': 8.23906024214195e-05}

 31%|███       | 3120/10000 [21:33<23:39,  4.85it/s]
 31%|███       | 3121/10000 [21:33<25:49,  4.44it/s]
 31%|███       | 3122/10000 [21:34<26:45,  4.29it/s]
 31%|███       | 3123/10000 [21:34<26:53,  4.26it/s]
 31%|███       | 3124/10000 [21:34<27:20,  4.19it/s]
 31%|███▏      | 3125/10000 [21:34<26:28,  4.33it/s]
 31%|███▏      | 3126/10000 [21:34<25:15,  4.54it/s]
 31%|███▏      | 3127/10000 [21:35<24:15,  4.72it/s]
 31%|███▏      | 3128/10000 [21:35<24:08,  4.74it/s]
 31%|███▏      | 3129/10000 [21:35<24:43,  4.63it/s]
 31%|███▏      | 3130/10000 [21:35<25:45,  4.45it/s]
                                                    
{'loss': 0.1109, 'grad_norm': 0.8495104908943176, 'learning_rate': 8.226446417743897e-05}

 31%|███▏      | 3130/10000 [21:35<25:45,  4.45it/s]
 31%|███▏      | 3131/10000 [21:36<28:43,  3.98it/s]
 31%|███▏      | 3132/10000 [21:36<28:42,  3.99it/s]
 31%|███▏      | 3133/10000 [21:36<26:30,  4.32it/s]
 31%|███▏      | 3134/10000 [21:36<25:24,  4.50it/s]
 31%|███▏      | 3135/10000 [21:36<24:25,  4.68it/s]
 31%|███▏      | 3136/10000 [21:37<24:04,  4.75it/s]
 31%|███▏      | 3137/10000 [21:37<24:33,  4.66it/s]
 31%|███▏      | 3138/10000 [21:37<25:52,  4.42it/s]
 31%|███▏      | 3139/10000 [21:37<27:40,  4.13it/s]
 31%|███▏      | 3140/10000 [21:38<27:57,  4.09it/s]
                                                    
{'loss': 0.117, 'grad_norm': 0.7839323878288269, 'learning_rate': 8.213797309444742e-05}

 31%|███▏      | 3140/10000 [21:38<27:57,  4.09it/s]
 31%|███▏      | 3141/10000 [21:38<27:36,  4.14it/s]
 31%|███▏      | 3142/10000 [21:38<26:40,  4.28it/s]
 31%|███▏      | 3143/10000 [21:38<25:53,  4.41it/s]
 31%|███▏      | 3144/10000 [21:39<24:59,  4.57it/s]
 31%|███▏      | 3145/10000 [21:39<25:03,  4.56it/s]
 31%|███▏      | 3146/10000 [21:39<25:21,  4.51it/s]
 31%|███▏      | 3147/10000 [21:39<25:51,  4.42it/s]
 31%|███▏      | 3148/10000 [21:39<25:50,  4.42it/s]
 31%|███▏      | 3149/10000 [21:40<25:14,  4.52it/s]
 32%|███▏      | 3150/10000 [21:40<25:23,  4.50it/s]
                                                    
{'loss': 0.1151, 'grad_norm': 0.9342600703239441, 'learning_rate': 8.201113055573105e-05}

 32%|███▏      | 3150/10000 [21:40<25:23,  4.50it/s]Rank 0, Worker 0: Wait for shard 21 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 32%|███▏      | 3151/10000 [21:40<25:02,  4.56it/s]
 32%|███▏      | 3152/10000 [21:40<24:38,  4.63it/s]
 32%|███▏      | 3153/10000 [21:40<24:10,  4.72it/s]
 32%|███▏      | 3154/10000 [21:41<24:05,  4.74it/s]
 32%|███▏      | 3155/10000 [21:41<24:27,  4.66it/s]
 32%|███▏      | 3156/10000 [21:41<25:23,  4.49it/s]
 32%|███▏      | 3157/10000 [21:41<25:22,  4.49it/s]
 32%|███▏      | 3158/10000 [21:42<25:05,  4.54it/s]
 32%|███▏      | 3159/10000 [21:42<25:30,  4.47it/s]
 32%|███▏      | 3160/10000 [21:42<24:57,  4.57it/s]
                                                    
{'loss': 0.0979, 'grad_norm': 1.1531493663787842, 'learning_rate': 8.188393794841958e-05}

 32%|███▏      | 3160/10000 [21:42<24:57,  4.57it/s]
 32%|███▏      | 3161/10000 [21:42<25:03,  4.55it/s]
 32%|███▏      | 3162/10000 [21:42<24:42,  4.61it/s]
 32%|███▏      | 3163/10000 [21:43<25:25,  4.48it/s]
 32%|███▏      | 3164/10000 [21:43<26:07,  4.36it/s]
 32%|███▏      | 3165/10000 [21:43<27:20,  4.17it/s]
 32%|███▏      | 3166/10000 [21:43<27:56,  4.08it/s]
 32%|███▏      | 3167/10000 [21:44<28:23,  4.01it/s]
 32%|███▏      | 3168/10000 [21:44<27:52,  4.08it/s]
 32%|███▏      | 3169/10000 [21:44<26:55,  4.23it/s]
 32%|███▏      | 3170/10000 [21:44<26:39,  4.27it/s]
                                                    
{'loss': 0.1482, 'grad_norm': 0.8980389833450317, 'learning_rate': 8.175639666347094e-05}

 32%|███▏      | 3170/10000 [21:44<26:39,  4.27it/s]
 32%|███▏      | 3171/10000 [21:45<26:50,  4.24it/s]
 32%|███▏      | 3172/10000 [21:45<26:19,  4.32it/s]
 32%|███▏      | 3173/10000 [21:45<25:58,  4.38it/s]
 32%|███▏      | 3174/10000 [21:45<26:09,  4.35it/s]
 32%|███▏      | 3175/10000 [21:46<25:35,  4.44it/s]
 32%|███▏      | 3176/10000 [21:46<25:02,  4.54it/s]
 32%|███▏      | 3177/10000 [21:46<24:53,  4.57it/s]
 32%|███▏      | 3178/10000 [21:46<24:21,  4.67it/s]
 32%|███▏      | 3179/10000 [21:46<23:51,  4.76it/s]
 32%|███▏      | 3180/10000 [21:47<23:43,  4.79it/s]
                                                    
{'loss': 0.1137, 'grad_norm': 0.8610150218009949, 'learning_rate': 8.162850809565623e-05}

 32%|███▏      | 3180/10000 [21:47<23:43,  4.79it/s]
 32%|███▏      | 3181/10000 [21:47<24:46,  4.59it/s]
 32%|███▏      | 3182/10000 [21:47<25:46,  4.41it/s]
 32%|███▏      | 3183/10000 [21:47<25:35,  4.44it/s]
 32%|███▏      | 3184/10000 [21:48<25:55,  4.38it/s]
 32%|███▏      | 3185/10000 [21:48<25:15,  4.50it/s]
 32%|███▏      | 3186/10000 [21:48<24:42,  4.60it/s]
 32%|███▏      | 3187/10000 [21:48<24:11,  4.69it/s]
 32%|███▏      | 3188/10000 [21:48<23:41,  4.79it/s]
 32%|███▏      | 3189/10000 [21:49<23:25,  4.85it/s]
 32%|███▏      | 3190/10000 [21:49<23:34,  4.81it/s]
                                                    
{'loss': 0.1221, 'grad_norm': 0.7497333884239197, 'learning_rate': 8.150027364354431e-05}

 32%|███▏      | 3190/10000 [21:49<23:34,  4.81it/s]
 32%|███▏      | 3191/10000 [21:49<24:09,  4.70it/s]
 32%|███▏      | 3192/10000 [21:49<24:04,  4.71it/s]
 32%|███▏      | 3193/10000 [21:49<25:42,  4.41it/s]
 32%|███▏      | 3194/10000 [21:50<26:23,  4.30it/s]
 32%|███▏      | 3195/10000 [21:50<26:43,  4.24it/s]
 32%|███▏      | 3196/10000 [21:50<25:39,  4.42it/s]
 32%|███▏      | 3197/10000 [21:50<24:25,  4.64it/s]
 32%|███▏      | 3198/10000 [21:51<24:06,  4.70it/s]
 32%|███▏      | 3199/10000 [21:51<24:08,  4.70it/s]
 32%|███▏      | 3200/10000 [21:51<24:24,  4.64it/s]
                                                    
{'loss': 0.134, 'grad_norm': 1.1180144548416138, 'learning_rate': 8.137169470948662e-05}

 32%|███▏      | 3200/10000 [21:51<24:24,  4.64it/s]
 32%|███▏      | 3201/10000 [21:51<25:26,  4.45it/s]
 32%|███▏      | 3202/10000 [21:51<25:59,  4.36it/s]
 32%|███▏      | 3203/10000 [21:52<25:50,  4.38it/s]
 32%|███▏      | 3204/10000 [21:52<25:33,  4.43it/s]
 32%|███▏      | 3205/10000 [21:52<25:10,  4.50it/s]
 32%|███▏      | 3206/10000 [21:52<24:33,  4.61it/s]
 32%|███▏      | 3207/10000 [21:53<24:12,  4.68it/s]
 32%|███▏      | 3208/10000 [21:53<24:02,  4.71it/s]
 32%|███▏      | 3209/10000 [21:53<23:49,  4.75it/s]
 32%|███▏      | 3210/10000 [21:53<23:51,  4.74it/s]
                                                    
{'loss': 0.1062, 'grad_norm': 1.0294067859649658, 'learning_rate': 8.124277269960179e-05}

 32%|███▏      | 3210/10000 [21:53<23:51,  4.74it/s]
 32%|███▏      | 3211/10000 [21:53<24:36,  4.60it/s]
 32%|███▏      | 3212/10000 [21:54<24:14,  4.67it/s]
 32%|███▏      | 3213/10000 [21:54<24:14,  4.66it/s]
 32%|███▏      | 3214/10000 [21:54<24:25,  4.63it/s]
 32%|███▏      | 3215/10000 [21:54<24:05,  4.69it/s]
 32%|███▏      | 3216/10000 [21:54<23:48,  4.75it/s]
 32%|███▏      | 3217/10000 [21:55<23:39,  4.78it/s]
 32%|███▏      | 3218/10000 [21:55<23:52,  4.73it/s]
 32%|███▏      | 3219/10000 [21:55<23:59,  4.71it/s]
 32%|███▏      | 3220/10000 [21:55<23:52,  4.73it/s]
                                                    
{'loss': 0.1265, 'grad_norm': 0.9504128098487854, 'learning_rate': 8.111350902376023e-05}

 32%|███▏      | 3220/10000 [21:55<23:52,  4.73it/s]
 32%|███▏      | 3221/10000 [21:55<23:57,  4.71it/s]
 32%|███▏      | 3222/10000 [21:56<23:38,  4.78it/s]
 32%|███▏      | 3223/10000 [21:56<23:27,  4.82it/s]
 32%|███▏      | 3224/10000 [21:56<23:27,  4.81it/s]
 32%|███▏      | 3225/10000 [21:56<24:01,  4.70it/s]
 32%|███▏      | 3226/10000 [21:57<24:14,  4.66it/s]
 32%|███▏      | 3227/10000 [21:57<23:58,  4.71it/s]
 32%|███▏      | 3228/10000 [21:57<23:36,  4.78it/s]
 32%|███▏      | 3229/10000 [21:57<22:57,  4.92it/s]
 32%|███▏      | 3230/10000 [21:57<22:33,  5.00it/s]
                                                    
{'loss': 0.1119, 'grad_norm': 1.010556936264038, 'learning_rate': 8.098390509556883e-05}

 32%|███▏      | 3230/10000 [21:57<22:33,  5.00it/s]
 32%|███▏      | 3231/10000 [21:58<22:52,  4.93it/s]
 32%|███▏      | 3232/10000 [21:58<23:09,  4.87it/s]
 32%|███▏      | 3233/10000 [21:58<23:21,  4.83it/s]
 32%|███▏      | 3234/10000 [21:58<23:19,  4.83it/s]
 32%|███▏      | 3235/10000 [21:58<23:00,  4.90it/s]
 32%|███▏      | 3236/10000 [21:59<22:14,  5.07it/s]
 32%|███▏      | 3237/10000 [21:59<21:36,  5.22it/s]
 32%|███▏      | 3238/10000 [21:59<21:08,  5.33it/s]
 32%|███▏      | 3239/10000 [21:59<21:26,  5.26it/s]
 32%|███▏      | 3240/10000 [21:59<21:50,  5.16it/s]
                                                    
{'loss': 0.1096, 'grad_norm': 1.0278180837631226, 'learning_rate': 8.085396233235536e-05}

 32%|███▏      | 3240/10000 [21:59<21:50,  5.16it/s]
 32%|███▏      | 3241/10000 [22:00<22:18,  5.05it/s]
 32%|███▏      | 3242/10000 [22:00<22:45,  4.95it/s]
 32%|███▏      | 3243/10000 [22:00<22:24,  5.03it/s]
 32%|███▏      | 3244/10000 [22:00<21:41,  5.19it/s]
 32%|███▏      | 3245/10000 [22:00<21:19,  5.28it/s]
 32%|███▏      | 3246/10000 [22:00<21:02,  5.35it/s]
 32%|███▏      | 3247/10000 [22:01<21:11,  5.31it/s]
 32%|███▏      | 3248/10000 [22:01<21:41,  5.19it/s]
 32%|███▏      | 3249/10000 [22:01<21:57,  5.12it/s]
 32%|███▎      | 3250/10000 [22:01<21:58,  5.12it/s]Rank 0, Worker 1: Caching shard...Rank 0, Worker 0: Caching shard...

Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.41 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 16.51 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 16.67 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.74 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 16.87 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 16.97 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 3250: eval/loss = 0.1222 (35s)

                                                    
{'loss': 0.1318, 'grad_norm': 0.6960654258728027, 'learning_rate': 8.072368215515306e-05}

 32%|███▎      | 3250/10000 [22:36<21:58,  5.12it/s]
 33%|███▎      | 3251/10000 [22:36<19:49:59, 10.58s/it]
 33%|███▎      | 3252/10000 [22:36<13:59:39,  7.47s/it]
 33%|███▎      | 3253/10000 [22:36<9:54:10,  5.28s/it] 
 33%|███▎      | 3254/10000 [22:37<7:01:54,  3.75s/it]
 33%|███▎      | 3255/10000 [22:37<5:01:16,  2.68s/it]
 33%|███▎      | 3256/10000 [22:37<3:36:50,  1.93s/it]
 33%|███▎      | 3257/10000 [22:37<2:38:06,  1.41s/it]
 33%|███▎      | 3258/10000 [22:37<1:57:21,  1.04s/it]
 33%|███▎      | 3259/10000 [22:38<1:28:53,  1.26it/s]
 33%|███▎      | 3260/10000 [22:38<1:08:59,  1.63it/s]
                                                      
{'loss': 0.1055, 'grad_norm': 0.7151311635971069, 'learning_rate': 8.059306598868506e-05}

 33%|███▎      | 3260/10000 [22:38<1:08:59,  1.63it/s]
 33%|███▎      | 3261/10000 [22:38<55:11,  2.03it/s]  
 33%|███▎      | 3262/10000 [22:38<44:37,  2.52it/s]
 33%|███▎      | 3263/10000 [22:38<37:29,  2.99it/s]
 33%|███▎      | 3264/10000 [22:39<32:40,  3.44it/s]
 33%|███▎      | 3265/10000 [22:39<29:33,  3.80it/s]
 33%|███▎      | 3266/10000 [22:39<27:20,  4.10it/s]
 33%|███▎      | 3267/10000 [22:39<25:51,  4.34it/s]
 33%|███▎      | 3268/10000 [22:39<24:39,  4.55it/s]
 33%|███▎      | 3269/10000 [22:40<23:49,  4.71it/s]
 33%|███▎      | 3270/10000 [22:40<23:35,  4.75it/s]
                                                    
{'loss': 0.1293, 'grad_norm': 0.9820314645767212, 'learning_rate': 8.046211526134888e-05}

 33%|███▎      | 3270/10000 [22:40<23:35,  4.75it/s]
 33%|███▎      | 3271/10000 [22:40<24:03,  4.66it/s]
 33%|███▎      | 3272/10000 [22:40<23:36,  4.75it/s]
 33%|███▎      | 3273/10000 [22:40<23:37,  4.75it/s]
 33%|███▎      | 3274/10000 [22:41<23:08,  4.84it/s]
 33%|███▎      | 3275/10000 [22:41<22:49,  4.91it/s]
 33%|███▎      | 3276/10000 [22:41<22:10,  5.06it/s]
 33%|███▎      | 3277/10000 [22:41<21:53,  5.12it/s]
 33%|███▎      | 3278/10000 [22:41<21:34,  5.19it/s]
 33%|███▎      | 3279/10000 [22:42<21:35,  5.19it/s]
 33%|███▎      | 3280/10000 [22:42<22:12,  5.04it/s]
                                                    
{'loss': 0.1228, 'grad_norm': 0.9405840039253235, 'learning_rate': 8.033083140520065e-05}

 33%|███▎      | 3280/10000 [22:42<22:12,  5.04it/s]
 33%|███▎      | 3281/10000 [22:42<23:16,  4.81it/s]
 33%|███▎      | 3282/10000 [22:42<23:55,  4.68it/s]
 33%|███▎      | 3283/10000 [22:42<23:54,  4.68it/s]
 33%|███▎      | 3284/10000 [22:43<23:55,  4.68it/s]
 33%|███▎      | 3285/10000 [22:43<23:35,  4.74it/s]
 33%|███▎      | 3286/10000 [22:43<23:14,  4.81it/s]
 33%|███▎      | 3287/10000 [22:43<23:20,  4.79it/s]
 33%|███▎      | 3288/10000 [22:43<23:50,  4.69it/s]
 33%|███▎      | 3289/10000 [22:44<23:11,  4.82it/s]
 33%|███▎      | 3290/10000 [22:44<23:08,  4.83it/s]
                                                    
{'loss': 0.1358, 'grad_norm': 0.9371180534362793, 'learning_rate': 8.019921585593962e-05}

 33%|███▎      | 3290/10000 [22:44<23:08,  4.83it/s]
 33%|███▎      | 3291/10000 [22:44<23:22,  4.79it/s]
 33%|███▎      | 3292/10000 [22:44<23:06,  4.84it/s]
 33%|███▎      | 3293/10000 [22:44<22:54,  4.88it/s]
 33%|███▎      | 3294/10000 [22:45<22:42,  4.92it/s]
 33%|███▎      | 3295/10000 [22:45<22:41,  4.93it/s]
 33%|███▎      | 3296/10000 [22:45<22:32,  4.96it/s]
 33%|███▎      | 3297/10000 [22:45<22:30,  4.96it/s]
 33%|███▎      | 3298/10000 [22:45<22:28,  4.97it/s]
 33%|███▎      | 3299/10000 [22:46<22:24,  4.98it/s]
 33%|███▎      | 3300/10000 [22:46<22:12,  5.03it/s]
                                                    
{'loss': 0.1264, 'grad_norm': 1.0023906230926514, 'learning_rate': 8.006727005289232e-05}

 33%|███▎      | 3300/10000 [22:46<22:12,  5.03it/s]
 33%|███▎      | 3301/10000 [22:46<22:24,  4.98it/s]
 33%|███▎      | 3302/10000 [22:46<22:34,  4.94it/s]
 33%|███▎      | 3303/10000 [22:46<22:39,  4.93it/s]
 33%|███▎      | 3304/10000 [22:47<22:56,  4.86it/s]
 33%|███▎      | 3305/10000 [22:47<22:39,  4.92it/s]
 33%|███▎      | 3306/10000 [22:47<22:18,  5.00it/s]
 33%|███▎      | 3307/10000 [22:47<21:41,  5.14it/s]
 33%|███▎      | 3308/10000 [22:47<21:13,  5.25it/s]
 33%|███▎      | 3309/10000 [22:48<20:55,  5.33it/s]
 33%|███▎      | 3310/10000 [22:48<22:15,  5.01it/s]
                                                    
{'loss': 0.1151, 'grad_norm': 0.8279885053634644, 'learning_rate': 7.993499543899692e-05}

 33%|███▎      | 3310/10000 [22:48<22:15,  5.01it/s]
 33%|███▎      | 3311/10000 [22:48<23:31,  4.74it/s]
 33%|███▎      | 3312/10000 [22:48<23:59,  4.65it/s]
 33%|███▎      | 3313/10000 [22:49<24:35,  4.53it/s]
 33%|███▎      | 3314/10000 [22:49<24:31,  4.54it/s]
 33%|███▎      | 3315/10000 [22:49<25:07,  4.43it/s]
 33%|███▎      | 3316/10000 [22:49<24:55,  4.47it/s]
 33%|███▎      | 3317/10000 [22:49<24:19,  4.58it/s]
 33%|███▎      | 3318/10000 [22:50<23:44,  4.69it/s]
 33%|███▎      | 3319/10000 [22:50<23:55,  4.65it/s]
 33%|███▎      | 3320/10000 [22:50<24:05,  4.62it/s]
                                                    
{'loss': 0.1238, 'grad_norm': 1.0948735475540161, 'learning_rate': 7.980239346078742e-05}

 33%|███▎      | 3320/10000 [22:50<24:05,  4.62it/s]
 33%|███▎      | 3321/10000 [22:50<24:50,  4.48it/s]
 33%|███▎      | 3322/10000 [22:51<24:59,  4.45it/s]
 33%|███▎      | 3323/10000 [22:51<24:08,  4.61it/s]
 33%|███▎      | 3324/10000 [22:51<23:42,  4.69it/s]
 33%|███▎      | 3325/10000 [22:51<22:58,  4.84it/s]
 33%|███▎      | 3326/10000 [22:51<22:33,  4.93it/s]
 33%|███▎      | 3327/10000 [22:51<21:44,  5.12it/s]
 33%|███▎      | 3328/10000 [22:52<21:36,  5.14it/s]
 33%|███▎      | 3329/10000 [22:52<21:22,  5.20it/s]
 33%|███▎      | 3330/10000 [22:52<21:03,  5.28it/s]
                                                    
{'loss': 0.1049, 'grad_norm': 1.0525538921356201, 'learning_rate': 7.966946556837778e-05}

 33%|███▎      | 3330/10000 [22:52<21:03,  5.28it/s]
 33%|███▎      | 3331/10000 [22:52<21:21,  5.20it/s]
 33%|███▎      | 3332/10000 [22:52<21:21,  5.20it/s]
 33%|███▎      | 3333/10000 [22:53<21:00,  5.29it/s]
 33%|███▎      | 3334/10000 [22:53<21:11,  5.24it/s]
 33%|███▎      | 3335/10000 [22:53<22:13,  5.00it/s]
 33%|███▎      | 3336/10000 [22:53<22:36,  4.91it/s]
 33%|███▎      | 3337/10000 [22:53<22:45,  4.88it/s]
 33%|███▎      | 3338/10000 [22:54<21:54,  5.07it/s]
 33%|███▎      | 3339/10000 [22:54<21:14,  5.23it/s]
 33%|███▎      | 3340/10000 [22:54<20:54,  5.31it/s]
                                                    
{'loss': 0.1375, 'grad_norm': 1.0738564729690552, 'learning_rate': 7.953621321544616e-05}

 33%|███▎      | 3340/10000 [22:54<20:54,  5.31it/s]
 33%|███▎      | 3341/10000 [22:54<20:59,  5.29it/s]
 33%|███▎      | 3342/10000 [22:54<20:59,  5.29it/s]
 33%|███▎      | 3343/10000 [22:55<21:13,  5.23it/s]
 33%|███▎      | 3344/10000 [22:55<21:17,  5.21it/s]
 33%|███▎      | 3345/10000 [22:55<21:15,  5.22it/s]
 33%|███▎      | 3346/10000 [22:55<21:20,  5.20it/s]
 33%|███▎      | 3347/10000 [22:55<20:52,  5.31it/s]
 33%|███▎      | 3348/10000 [22:56<20:52,  5.31it/s]
 33%|███▎      | 3349/10000 [22:56<20:46,  5.34it/s]
 34%|███▎      | 3350/10000 [22:56<20:27,  5.42it/s]
                                                    
{'loss': 0.121, 'grad_norm': 0.9360809326171875, 'learning_rate': 7.940263785921896e-05}

 34%|███▎      | 3350/10000 [22:56<20:27,  5.42it/s]
 34%|███▎      | 3351/10000 [22:56<20:38,  5.37it/s]
 34%|███▎      | 3352/10000 [22:56<20:25,  5.43it/s]
 34%|███▎      | 3353/10000 [22:56<20:30,  5.40it/s]
 34%|███▎      | 3354/10000 [22:57<20:38,  5.37it/s]
 34%|███▎      | 3355/10000 [22:57<20:53,  5.30it/s]
 34%|███▎      | 3356/10000 [22:57<20:55,  5.29it/s]
 34%|███▎      | 3357/10000 [22:57<20:40,  5.36it/s]
 34%|███▎      | 3358/10000 [22:57<20:37,  5.37it/s]
 34%|███▎      | 3359/10000 [22:58<20:50,  5.31it/s]
 34%|███▎      | 3360/10000 [22:58<20:53,  5.30it/s]
                                                    
{'loss': 0.1143, 'grad_norm': 0.9762896299362183, 'learning_rate': 7.926874096045482e-05}

 34%|███▎      | 3360/10000 [22:58<20:53,  5.30it/s]
 34%|███▎      | 3361/10000 [22:58<22:32,  4.91it/s]
 34%|███▎      | 3362/10000 [22:58<21:40,  5.10it/s]
 34%|███▎      | 3363/10000 [22:58<21:12,  5.21it/s]
 34%|███▎      | 3364/10000 [22:59<20:49,  5.31it/s]
 34%|███▎      | 3365/10000 [22:59<20:26,  5.41it/s]
 34%|███▎      | 3366/10000 [22:59<20:09,  5.49it/s]
 34%|███▎      | 3367/10000 [22:59<20:35,  5.37it/s]
 34%|███▎      | 3368/10000 [22:59<20:37,  5.36it/s]
 34%|███▎      | 3369/10000 [22:59<20:41,  5.34it/s]
 34%|███▎      | 3370/10000 [23:00<21:48,  5.07it/s]
                                                    
{'loss': 0.1239, 'grad_norm': 1.1752498149871826, 'learning_rate': 7.913452398342881e-05}

 34%|███▎      | 3370/10000 [23:00<21:48,  5.07it/s]
 34%|███▎      | 3371/10000 [23:00<22:27,  4.92it/s]
 34%|███▎      | 3372/10000 [23:00<21:32,  5.13it/s]
 34%|███▎      | 3373/10000 [23:00<21:14,  5.20it/s]
 34%|███▎      | 3374/10000 [23:00<20:58,  5.27it/s]
 34%|███▍      | 3375/10000 [23:01<20:45,  5.32it/s]
 34%|███▍      | 3376/10000 [23:01<20:58,  5.26it/s]
 34%|███▍      | 3377/10000 [23:01<21:22,  5.16it/s]
 34%|███▍      | 3378/10000 [23:01<21:36,  5.11it/s]
 34%|███▍      | 3379/10000 [23:01<22:18,  4.95it/s]
 34%|███▍      | 3380/10000 [23:02<21:31,  5.12it/s]
                                                    
{'loss': 0.124, 'grad_norm': 1.0548765659332275, 'learning_rate': 7.89999883959163e-05}

 34%|███▍      | 3380/10000 [23:02<21:31,  5.12it/s]
 34%|███▍      | 3381/10000 [23:02<21:23,  5.16it/s]
 34%|███▍      | 3382/10000 [23:02<21:23,  5.15it/s]
 34%|███▍      | 3383/10000 [23:02<21:34,  5.11it/s]
 34%|███▍      | 3384/10000 [23:02<21:52,  5.04it/s]
 34%|███▍      | 3385/10000 [23:03<22:12,  4.96it/s]
 34%|███▍      | 3386/10000 [23:03<22:17,  4.95it/s]
 34%|███▍      | 3387/10000 [23:03<22:09,  4.97it/s]
 34%|███▍      | 3388/10000 [23:03<22:02,  5.00it/s]
 34%|███▍      | 3389/10000 [23:03<21:33,  5.11it/s]
 34%|███▍      | 3390/10000 [23:04<21:57,  5.02it/s]
                                                    
{'loss': 0.0932, 'grad_norm': 0.670444905757904, 'learning_rate': 7.886513566917687e-05}

 34%|███▍      | 3390/10000 [23:04<21:57,  5.02it/s]
 34%|███▍      | 3391/10000 [23:04<22:16,  4.94it/s]
 34%|███▍      | 3392/10000 [23:04<22:09,  4.97it/s]
 34%|███▍      | 3393/10000 [23:04<22:41,  4.85it/s]
 34%|███▍      | 3394/10000 [23:04<23:00,  4.79it/s]
 34%|███▍      | 3395/10000 [23:05<22:33,  4.88it/s]
 34%|███▍      | 3396/10000 [23:05<22:10,  4.96it/s]
 34%|███▍      | 3397/10000 [23:05<21:41,  5.07it/s]
 34%|███▍      | 3398/10000 [23:05<21:50,  5.04it/s]
 34%|███▍      | 3399/10000 [23:05<22:16,  4.94it/s]
 34%|███▍      | 3400/10000 [23:06<22:24,  4.91it/s]
                                                    
{'loss': 0.128, 'grad_norm': 1.0821609497070312, 'learning_rate': 7.872996727793838e-05}

 34%|███▍      | 3400/10000 [23:06<22:24,  4.91it/s]
 34%|███▍      | 3401/10000 [23:06<22:53,  4.80it/s]
 34%|███▍      | 3402/10000 [23:06<22:54,  4.80it/s]
 34%|███▍      | 3403/10000 [23:06<22:39,  4.85it/s]
 34%|███▍      | 3404/10000 [23:07<23:00,  4.78it/s]
 34%|███▍      | 3405/10000 [23:07<23:12,  4.74it/s]
 34%|███▍      | 3406/10000 [23:07<22:48,  4.82it/s]
 34%|███▍      | 3407/10000 [23:07<22:41,  4.84it/s]
 34%|███▍      | 3408/10000 [23:07<22:34,  4.87it/s]
 34%|███▍      | 3409/10000 [23:08<22:03,  4.98it/s]
 34%|███▍      | 3410/10000 [23:08<21:50,  5.03it/s]
                                                    
{'loss': 0.1146, 'grad_norm': 1.0120450258255005, 'learning_rate': 7.859448470038069e-05}

 34%|███▍      | 3410/10000 [23:08<21:50,  5.03it/s]
 34%|███▍      | 3411/10000 [23:08<21:35,  5.09it/s]
 34%|███▍      | 3412/10000 [23:08<21:25,  5.13it/s]
 34%|███▍      | 3413/10000 [23:08<21:21,  5.14it/s]
 34%|███▍      | 3414/10000 [23:08<21:14,  5.17it/s]
 34%|███▍      | 3415/10000 [23:09<21:13,  5.17it/s]
 34%|███▍      | 3416/10000 [23:09<21:19,  5.14it/s]
 34%|███▍      | 3417/10000 [23:09<21:19,  5.14it/s]
 34%|███▍      | 3418/10000 [23:09<21:21,  5.14it/s]
 34%|███▍      | 3419/10000 [23:09<21:27,  5.11it/s]
 34%|███▍      | 3420/10000 [23:10<21:26,  5.12it/s]
                                                    
{'loss': 0.1189, 'grad_norm': 0.856132447719574, 'learning_rate': 7.845868941811956e-05}

 34%|███▍      | 3420/10000 [23:10<21:26,  5.12it/s]
 34%|███▍      | 3421/10000 [23:10<21:45,  5.04it/s]
 34%|███▍      | 3422/10000 [23:10<21:41,  5.05it/s]
 34%|███▍      | 3423/10000 [23:10<21:35,  5.08it/s]
 34%|███▍      | 3424/10000 [23:10<20:59,  5.22it/s]
 34%|███▍      | 3425/10000 [23:11<21:08,  5.18it/s]
 34%|███▍      | 3426/10000 [23:11<21:04,  5.20it/s]
 34%|███▍      | 3427/10000 [23:11<21:08,  5.18it/s]
 34%|███▍      | 3428/10000 [23:11<21:23,  5.12it/s]
 34%|███▍      | 3429/10000 [23:11<21:15,  5.15it/s]
 34%|███▍      | 3430/10000 [23:12<21:37,  5.06it/s]
                                                    
{'loss': 0.1111, 'grad_norm': 0.8527117967605591, 'learning_rate': 7.832258291619043e-05}

 34%|███▍      | 3430/10000 [23:12<21:37,  5.06it/s]
 34%|███▍      | 3431/10000 [23:12<22:01,  4.97it/s]
 34%|███▍      | 3432/10000 [23:12<21:53,  5.00it/s]
 34%|███▍      | 3433/10000 [23:12<22:03,  4.96it/s]
 34%|███▍      | 3434/10000 [23:12<22:00,  4.97it/s]
 34%|███▍      | 3435/10000 [23:13<21:51,  5.01it/s]
 34%|███▍      | 3436/10000 [23:13<21:50,  5.01it/s]
 34%|███▍      | 3437/10000 [23:13<21:32,  5.08it/s]
 34%|███▍      | 3438/10000 [23:13<21:21,  5.12it/s]
 34%|███▍      | 3439/10000 [23:13<22:19,  4.90it/s]
 34%|███▍      | 3440/10000 [23:14<22:02,  4.96it/s]
                                                    
{'loss': 0.1071, 'grad_norm': 0.9233558177947998, 'learning_rate': 7.81861666830322e-05}

 34%|███▍      | 3440/10000 [23:14<22:02,  4.96it/s]
 34%|███▍      | 3441/10000 [23:14<22:11,  4.93it/s]
 34%|███▍      | 3442/10000 [23:14<22:03,  4.96it/s]
 34%|███▍      | 3443/10000 [23:14<21:45,  5.02it/s]
 34%|███▍      | 3444/10000 [23:14<21:17,  5.13it/s]
 34%|███▍      | 3445/10000 [23:15<21:05,  5.18it/s]
 34%|███▍      | 3446/10000 [23:15<20:39,  5.29it/s]
 34%|███▍      | 3447/10000 [23:15<20:35,  5.30it/s]
 34%|███▍      | 3448/10000 [23:15<20:38,  5.29it/s]
 34%|███▍      | 3449/10000 [23:15<20:41,  5.28it/s]
 34%|███▍      | 3450/10000 [23:16<20:53,  5.23it/s]
                                                    
{'loss': 0.097, 'grad_norm': 0.8561029434204102, 'learning_rate': 7.804944221047097e-05}

 34%|███▍      | 3450/10000 [23:16<20:53,  5.23it/s]
 35%|███▍      | 3451/10000 [23:16<21:21,  5.11it/s]
 35%|███▍      | 3452/10000 [23:16<21:28,  5.08it/s]
 35%|███▍      | 3453/10000 [23:16<21:17,  5.13it/s]
 35%|███▍      | 3454/10000 [23:16<21:05,  5.17it/s]
 35%|███▍      | 3455/10000 [23:16<20:38,  5.28it/s]
 35%|███▍      | 3456/10000 [23:17<20:09,  5.41it/s]
 35%|███▍      | 3457/10000 [23:17<20:37,  5.29it/s]
 35%|███▍      | 3458/10000 [23:17<20:12,  5.39it/s]
 35%|███▍      | 3459/10000 [23:17<20:16,  5.38it/s]
 35%|███▍      | 3460/10000 [23:17<20:28,  5.32it/s]
                                                    
{'loss': 0.1255, 'grad_norm': 0.9383890628814697, 'learning_rate': 7.791241099370364e-05}

 35%|███▍      | 3460/10000 [23:17<20:28,  5.32it/s]
 35%|███▍      | 3461/10000 [23:18<20:50,  5.23it/s]
 35%|███▍      | 3462/10000 [23:18<20:41,  5.27it/s]
 35%|███▍      | 3463/10000 [23:18<20:20,  5.36it/s]
 35%|███▍      | 3464/10000 [23:18<19:53,  5.48it/s]
 35%|███▍      | 3465/10000 [23:18<19:41,  5.53it/s]
 35%|███▍      | 3466/10000 [23:19<19:30,  5.58it/s]
 35%|███▍      | 3467/10000 [23:19<19:21,  5.62it/s]
 35%|███▍      | 3468/10000 [23:19<19:33,  5.57it/s]
 35%|███▍      | 3469/10000 [23:19<20:05,  5.42it/s]
 35%|███▍      | 3470/10000 [23:19<20:17,  5.36it/s]
                                                    
{'loss': 0.1183, 'grad_norm': 0.9625961780548096, 'learning_rate': 7.777507453128163e-05}

 35%|███▍      | 3470/10000 [23:19<20:17,  5.36it/s]
 35%|███▍      | 3471/10000 [23:19<20:26,  5.32it/s]
 35%|███▍      | 3472/10000 [23:20<20:23,  5.33it/s]
 35%|███▍      | 3473/10000 [23:20<20:06,  5.41it/s]
 35%|███▍      | 3474/10000 [23:20<21:20,  5.10it/s]
 35%|███▍      | 3475/10000 [23:20<22:16,  4.88it/s]
 35%|███▍      | 3476/10000 [23:21<23:33,  4.62it/s]
 35%|███▍      | 3477/10000 [23:21<24:13,  4.49it/s]
 35%|███▍      | 3478/10000 [23:21<24:18,  4.47it/s]
 35%|███▍      | 3479/10000 [23:21<23:03,  4.72it/s]
 35%|███▍      | 3480/10000 [23:21<22:09,  4.90it/s]
                                                    
{'loss': 0.1187, 'grad_norm': 1.0405118465423584, 'learning_rate': 7.763743432509451e-05}

 35%|███▍      | 3480/10000 [23:21<22:09,  4.90it/s]
 35%|███▍      | 3481/10000 [23:22<21:46,  4.99it/s]
 35%|███▍      | 3482/10000 [23:22<21:25,  5.07it/s]
 35%|███▍      | 3483/10000 [23:22<23:48,  4.56it/s]
 35%|███▍      | 3484/10000 [23:22<24:01,  4.52it/s]
 35%|███▍      | 3485/10000 [23:22<24:05,  4.51it/s]
 35%|███▍      | 3486/10000 [23:23<24:09,  4.49it/s]
 35%|███▍      | 3487/10000 [23:23<23:40,  4.58it/s]
 35%|███▍      | 3488/10000 [23:23<22:37,  4.80it/s]
 35%|███▍      | 3489/10000 [23:23<22:09,  4.90it/s]
 35%|███▍      | 3490/10000 [23:23<22:07,  4.90it/s]
                                                    
{'loss': 0.1205, 'grad_norm': 0.9360929131507874, 'learning_rate': 7.749949188035353e-05}

 35%|███▍      | 3490/10000 [23:24<22:07,  4.90it/s]
 35%|███▍      | 3491/10000 [23:24<22:28,  4.83it/s]
 35%|███▍      | 3492/10000 [23:24<22:28,  4.82it/s]
 35%|███▍      | 3493/10000 [23:24<23:44,  4.57it/s]
 35%|███▍      | 3494/10000 [23:24<24:04,  4.50it/s]
 35%|███▍      | 3495/10000 [23:25<24:03,  4.51it/s]
 35%|███▍      | 3496/10000 [23:25<23:25,  4.63it/s]
 35%|███▍      | 3497/10000 [23:25<22:44,  4.77it/s]
 35%|███▍      | 3498/10000 [23:25<21:47,  4.97it/s]
 35%|███▍      | 3499/10000 [23:25<21:22,  5.07it/s]
 35%|███▌      | 3500/10000 [23:26<22:35,  4.80it/s]Rank 0, Worker 1: Caching shard...Rank 0, Worker 0: Caching shard...

Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 4: Caching shard...

Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.38 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.00 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.10 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.62 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.65 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.67 seconds
Rank 0, Worker 0: Caching shard...
[held-out-eval] step 3500: eval/loss = 0.1139 (38s)

                                                    
{'loss': 0.1344, 'grad_norm': 0.9875292181968689, 'learning_rate': 7.736124870557516e-05}

 35%|███▌      | 3500/10000 [24:04<22:35,  4.80it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-3500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-3500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-3500/wandb_config.json

 35%|███▌      | 3501/10000 [24:32<36:19:01, 20.12s/it]
 35%|███▌      | 3502/10000 [24:32<25:32:10, 14.15s/it]
 35%|███▌      | 3503/10000 [24:33<17:59:15,  9.97s/it]
 35%|███▌      | 3504/10000 [24:33<12:41:27,  7.03s/it]
 35%|███▌      | 3505/10000 [24:33<8:58:58,  4.98s/it] 
 35%|███▌      | 3506/10000 [24:33<6:23:10,  3.54s/it]
 35%|███▌      | 3507/10000 [24:33<4:34:30,  2.54s/it]
 35%|███▌      | 3508/10000 [24:34<3:18:53,  1.84s/it]
 35%|███▌      | 3509/10000 [24:34<2:25:47,  1.35s/it]
 35%|███▌      | 3510/10000 [24:34<1:48:52,  1.01s/it]
                                                      
{'loss': 0.1154, 'grad_norm': 0.917805016040802, 'learning_rate': 7.722270631256459e-05}

 35%|███▌      | 3510/10000 [24:34<1:48:52,  1.01s/it]
 35%|███▌      | 3511/10000 [24:34<1:23:44,  1.29it/s]
 35%|███▌      | 3512/10000 [24:34<1:05:38,  1.65it/s]
 35%|███▌      | 3513/10000 [24:35<52:20,  2.07it/s]  
 35%|███▌      | 3514/10000 [24:35<42:25,  2.55it/s]
 35%|███▌      | 3515/10000 [24:35<35:47,  3.02it/s]
 35%|███▌      | 3516/10000 [24:35<31:02,  3.48it/s]
 35%|███▌      | 3517/10000 [24:35<28:10,  3.84it/s]
 35%|███▌      | 3518/10000 [24:36<26:17,  4.11it/s]
 35%|███▌      | 3519/10000 [24:36<25:00,  4.32it/s]
 35%|███▌      | 3520/10000 [24:36<24:19,  4.44it/s]
                                                    
{'loss': 0.1281, 'grad_norm': 0.9355779886245728, 'learning_rate': 7.708386621639925e-05}

 35%|███▌      | 3520/10000 [24:36<24:19,  4.44it/s]
 35%|███▌      | 3521/10000 [24:36<23:56,  4.51it/s]
 35%|███▌      | 3522/10000 [24:36<23:02,  4.69it/s]
 35%|███▌      | 3523/10000 [24:37<21:56,  4.92it/s]
 35%|███▌      | 3524/10000 [24:37<21:28,  5.03it/s]
 35%|███▌      | 3525/10000 [24:37<20:53,  5.16it/s]
 35%|███▌      | 3526/10000 [24:37<20:53,  5.17it/s]
 35%|███▌      | 3527/10000 [24:37<21:18,  5.06it/s]
 35%|███▌      | 3528/10000 [24:38<21:47,  4.95it/s]
 35%|███▌      | 3529/10000 [24:38<22:14,  4.85it/s]
 35%|███▌      | 3530/10000 [24:38<22:54,  4.71it/s]
                                                    
{'loss': 0.1466, 'grad_norm': 0.8040031790733337, 'learning_rate': 7.694472993541219e-05}

 35%|███▌      | 3530/10000 [24:38<22:54,  4.71it/s]
 35%|███▌      | 3531/10000 [24:38<22:55,  4.70it/s]
 35%|███▌      | 3532/10000 [24:38<22:08,  4.87it/s]
 35%|███▌      | 3533/10000 [24:39<21:31,  5.01it/s]
 35%|███▌      | 3534/10000 [24:39<21:00,  5.13it/s]
 35%|███▌      | 3535/10000 [24:39<21:12,  5.08it/s]
 35%|███▌      | 3536/10000 [24:39<21:21,  5.04it/s]
 35%|███▌      | 3537/10000 [24:39<21:32,  5.00it/s]
 35%|███▌      | 3538/10000 [24:40<22:03,  4.88it/s]
 35%|███▌      | 3539/10000 [24:40<22:06,  4.87it/s]
 35%|███▌      | 3540/10000 [24:40<23:14,  4.63it/s]
                                                    
{'loss': 0.1243, 'grad_norm': 1.2459099292755127, 'learning_rate': 7.680529899117547e-05}

 35%|███▌      | 3540/10000 [24:40<23:14,  4.63it/s]
 35%|███▌      | 3541/10000 [24:40<22:35,  4.77it/s]
 35%|███▌      | 3542/10000 [24:40<22:18,  4.83it/s]
 35%|███▌      | 3543/10000 [24:41<22:00,  4.89it/s]
 35%|███▌      | 3544/10000 [24:41<21:59,  4.89it/s]
 35%|███▌      | 3545/10000 [24:41<22:04,  4.87it/s]
 35%|███▌      | 3546/10000 [24:41<22:25,  4.80it/s]
 35%|███▌      | 3547/10000 [24:41<22:40,  4.74it/s]
 35%|███▌      | 3548/10000 [24:42<22:03,  4.88it/s]
 35%|███▌      | 3549/10000 [24:42<21:15,  5.06it/s]
 36%|███▌      | 3550/10000 [24:42<20:41,  5.20it/s]
                                                    
{'loss': 0.1369, 'grad_norm': 0.9648393392562866, 'learning_rate': 7.666557490848358e-05}

 36%|███▌      | 3550/10000 [24:42<20:41,  5.20it/s]
 36%|███▌      | 3551/10000 [24:42<21:06,  5.09it/s]
 36%|███▌      | 3552/10000 [24:42<21:38,  4.97it/s]
 36%|███▌      | 3553/10000 [24:43<22:08,  4.85it/s]
 36%|███▌      | 3554/10000 [24:43<22:16,  4.82it/s]
 36%|███▌      | 3555/10000 [24:43<22:31,  4.77it/s]
 36%|███▌      | 3556/10000 [24:43<22:02,  4.87it/s]
 36%|███▌      | 3557/10000 [24:43<21:06,  5.09it/s]
 36%|███▌      | 3558/10000 [24:44<20:42,  5.18it/s]
 36%|███▌      | 3559/10000 [24:44<20:40,  5.19it/s]
 36%|███▌      | 3560/10000 [24:44<21:02,  5.10it/s]
                                                    
{'loss': 0.1155, 'grad_norm': 0.9019229412078857, 'learning_rate': 7.65255592153367e-05}

 36%|███▌      | 3560/10000 [24:44<21:02,  5.10it/s]
 36%|███▌      | 3561/10000 [24:44<21:53,  4.90it/s]
 36%|███▌      | 3562/10000 [24:44<22:27,  4.78it/s]
 36%|███▌      | 3563/10000 [24:45<22:35,  4.75it/s]
 36%|███▌      | 3564/10000 [24:45<21:52,  4.90it/s]
 36%|███▌      | 3565/10000 [24:45<21:02,  5.10it/s]
 36%|███▌      | 3566/10000 [24:45<20:31,  5.23it/s]
 36%|███▌      | 3567/10000 [24:45<20:27,  5.24it/s]
 36%|███▌      | 3568/10000 [24:46<20:27,  5.24it/s]
 36%|███▌      | 3569/10000 [24:46<21:03,  5.09it/s]
 36%|███▌      | 3570/10000 [24:46<21:41,  4.94it/s]
                                                    
{'loss': 0.1198, 'grad_norm': 0.8202961087226868, 'learning_rate': 7.638525344292402e-05}

 36%|███▌      | 3570/10000 [24:46<21:41,  4.94it/s]
 36%|███▌      | 3571/10000 [24:46<21:57,  4.88it/s]
 36%|███▌      | 3572/10000 [24:46<21:16,  5.04it/s]
 36%|███▌      | 3573/10000 [24:47<20:49,  5.14it/s]
 36%|███▌      | 3574/10000 [24:47<20:21,  5.26it/s]
 36%|███▌      | 3575/10000 [24:47<20:05,  5.33it/s]
 36%|███▌      | 3576/10000 [24:47<19:47,  5.41it/s]
 36%|███▌      | 3577/10000 [24:47<19:28,  5.50it/s]
 36%|███▌      | 3578/10000 [24:48<19:28,  5.50it/s]
 36%|███▌      | 3579/10000 [24:48<19:32,  5.48it/s]
 36%|███▌      | 3580/10000 [24:48<20:32,  5.21it/s]
                                                    
{'loss': 0.1071, 'grad_norm': 0.9881982803344727, 'learning_rate': 7.624465912560697e-05}

 36%|███▌      | 3580/10000 [24:48<20:32,  5.21it/s]
 36%|███▌      | 3581/10000 [24:48<22:06,  4.84it/s]
 36%|███▌      | 3582/10000 [24:48<22:39,  4.72it/s]
 36%|███▌      | 3583/10000 [24:49<23:32,  4.54it/s]
 36%|███▌      | 3584/10000 [24:49<24:23,  4.38it/s]
 36%|███▌      | 3585/10000 [24:49<23:35,  4.53it/s]
 36%|███▌      | 3586/10000 [24:49<22:19,  4.79it/s]
 36%|███▌      | 3587/10000 [24:49<21:53,  4.88it/s]
 36%|███▌      | 3588/10000 [24:50<21:36,  4.95it/s]
 36%|███▌      | 3589/10000 [24:50<21:36,  4.94it/s]
 36%|███▌      | 3590/10000 [24:50<22:54,  4.66it/s]
                                                    
{'loss': 0.1037, 'grad_norm': 0.7380572557449341, 'learning_rate': 7.610377780090249e-05}

 36%|███▌      | 3590/10000 [24:50<22:54,  4.66it/s]
 36%|███▌      | 3591/10000 [24:50<24:21,  4.39it/s]
 36%|███▌      | 3592/10000 [24:51<24:10,  4.42it/s]
 36%|███▌      | 3593/10000 [24:51<24:34,  4.34it/s]
 36%|███▌      | 3594/10000 [24:51<24:55,  4.28it/s]
 36%|███▌      | 3595/10000 [24:51<23:57,  4.45it/s]
 36%|███▌      | 3596/10000 [24:51<23:21,  4.57it/s]
 36%|███▌      | 3597/10000 [24:52<23:12,  4.60it/s]
 36%|███▌      | 3598/10000 [24:52<23:14,  4.59it/s]
 36%|███▌      | 3599/10000 [24:52<23:21,  4.57it/s]
 36%|███▌      | 3600/10000 [24:52<23:48,  4.48it/s]
                                                    
{'loss': 0.0933, 'grad_norm': 0.7854310274124146, 'learning_rate': 7.596261100946618e-05}

 36%|███▌      | 3600/10000 [24:52<23:48,  4.48it/s]
 36%|███▌      | 3601/10000 [24:53<26:04,  4.09it/s]
 36%|███▌      | 3602/10000 [24:53<26:37,  4.01it/s]
 36%|███▌      | 3603/10000 [24:53<27:03,  3.94it/s]
 36%|███▌      | 3604/10000 [24:53<26:46,  3.98it/s]
 36%|███▌      | 3605/10000 [24:54<25:02,  4.25it/s]
 36%|███▌      | 3606/10000 [24:54<23:27,  4.54it/s]
 36%|███▌      | 3607/10000 [24:54<22:49,  4.67it/s]
 36%|███▌      | 3608/10000 [24:54<21:52,  4.87it/s]
 36%|███▌      | 3609/10000 [24:54<22:30,  4.73it/s]
 36%|███▌      | 3610/10000 [24:55<23:08,  4.60it/s]
                                                    
{'loss': 0.1182, 'grad_norm': 1.0953775644302368, 'learning_rate': 7.582116029507542e-05}

 36%|███▌      | 3610/10000 [24:55<23:08,  4.60it/s]
 36%|███▌      | 3611/10000 [24:55<23:49,  4.47it/s]
 36%|███▌      | 3612/10000 [24:55<25:10,  4.23it/s]
 36%|███▌      | 3613/10000 [24:55<27:33,  3.86it/s]
 36%|███▌      | 3614/10000 [24:56<26:55,  3.95it/s]
 36%|███▌      | 3615/10000 [24:56<25:19,  4.20it/s]
 36%|███▌      | 3616/10000 [24:56<24:12,  4.40it/s]
 36%|███▌      | 3617/10000 [24:56<23:42,  4.49it/s]
 36%|███▌      | 3618/10000 [24:57<23:33,  4.52it/s]
 36%|███▌      | 3619/10000 [24:57<23:24,  4.54it/s]
 36%|███▌      | 3620/10000 [24:57<24:05,  4.42it/s]
                                                    
{'loss': 0.1425, 'grad_norm': 0.8806854486465454, 'learning_rate': 7.56794272046126e-05}

 36%|███▌      | 3620/10000 [24:57<24:05,  4.42it/s]
 36%|███▌      | 3621/10000 [24:57<25:13,  4.22it/s]
 36%|███▌      | 3622/10000 [24:58<25:43,  4.13it/s]
 36%|███▌      | 3623/10000 [24:58<24:27,  4.34it/s]
 36%|███▌      | 3624/10000 [24:58<23:27,  4.53it/s]
 36%|███▋      | 3625/10000 [24:58<22:45,  4.67it/s]
 36%|███▋      | 3626/10000 [24:58<22:35,  4.70it/s]
 36%|███▋      | 3627/10000 [24:59<24:02,  4.42it/s]
 36%|███▋      | 3628/10000 [24:59<23:37,  4.50it/s]
 36%|███▋      | 3629/10000 [24:59<24:19,  4.36it/s]
 36%|███▋      | 3630/10000 [24:59<24:20,  4.36it/s]
                                                    
{'loss': 0.0969, 'grad_norm': 0.853931188583374, 'learning_rate': 7.55374132880481e-05}

 36%|███▋      | 3630/10000 [24:59<24:20,  4.36it/s]
 36%|███▋      | 3631/10000 [24:59<23:22,  4.54it/s]
 36%|███▋      | 3632/10000 [25:00<22:42,  4.67it/s]
 36%|███▋      | 3633/10000 [25:00<22:22,  4.74it/s]
 36%|███▋      | 3634/10000 [25:00<22:15,  4.77it/s]
 36%|███▋      | 3635/10000 [25:00<22:29,  4.72it/s]
 36%|███▋      | 3636/10000 [25:01<29:15,  3.63it/s]
 36%|███▋      | 3637/10000 [25:01<28:12,  3.76it/s]
 36%|███▋      | 3638/10000 [25:01<35:53,  2.95it/s]
 36%|███▋      | 3639/10000 [25:02<31:36,  3.35it/s]
 36%|███▋      | 3640/10000 [25:02<28:40,  3.70it/s]
                                                    
{'loss': 0.1014, 'grad_norm': 0.8848007321357727, 'learning_rate': 7.539512009842333e-05}

 36%|███▋      | 3640/10000 [25:02<28:40,  3.70it/s]
 36%|███▋      | 3641/10000 [25:02<27:09,  3.90it/s]
 36%|███▋      | 3642/10000 [25:02<26:01,  4.07it/s]
 36%|███▋      | 3643/10000 [25:03<25:34,  4.14it/s]
 36%|███▋      | 3644/10000 [25:03<24:44,  4.28it/s]
 36%|███▋      | 3645/10000 [25:03<23:33,  4.50it/s]
 36%|███▋      | 3646/10000 [25:03<22:36,  4.68it/s]
 36%|███▋      | 3647/10000 [25:03<22:01,  4.81it/s]
 36%|███▋      | 3648/10000 [25:04<21:50,  4.85it/s]
 36%|███▋      | 3649/10000 [25:04<21:46,  4.86it/s]
 36%|███▋      | 3650/10000 [25:04<21:47,  4.86it/s]
                                                    
{'loss': 0.0897, 'grad_norm': 0.7920210957527161, 'learning_rate': 7.525254919183382e-05}

 36%|███▋      | 3650/10000 [25:04<21:47,  4.86it/s]
 37%|███▋      | 3651/10000 [25:04<22:06,  4.79it/s]
 37%|███▋      | 3652/10000 [25:04<21:55,  4.82it/s]Rank 0, Worker 4: Wait for shard 51 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 37%|███▋      | 3653/10000 [25:05<21:56,  4.82it/s]
 37%|███▋      | 3654/10000 [25:05<21:49,  4.84it/s]
 37%|███▋      | 3655/10000 [25:05<21:41,  4.88it/s]
 37%|███▋      | 3656/10000 [25:05<21:28,  4.92it/s]
 37%|███▋      | 3657/10000 [25:05<21:19,  4.96it/s]
 37%|███▋      | 3658/10000 [25:06<21:34,  4.90it/s]
 37%|███▋      | 3659/10000 [25:06<21:46,  4.85it/s]
 37%|███▋      | 3660/10000 [25:06<21:53,  4.83it/s]
                                                    
{'loss': 0.0995, 'grad_norm': 0.8340643644332886, 'learning_rate': 7.510970212741215e-05}

 37%|███▋      | 3660/10000 [25:06<21:53,  4.83it/s]
 37%|███▋      | 3661/10000 [25:06<22:38,  4.67it/s]
 37%|███▋      | 3662/10000 [25:06<22:35,  4.67it/s]
 37%|███▋      | 3663/10000 [25:07<22:17,  4.74it/s]
 37%|███▋      | 3664/10000 [25:07<22:11,  4.76it/s]
 37%|███▋      | 3665/10000 [25:07<21:51,  4.83it/s]
 37%|███▋      | 3666/10000 [25:07<21:01,  5.02it/s]
 37%|███▋      | 3667/10000 [25:07<20:34,  5.13it/s]
 37%|███▋      | 3668/10000 [25:08<20:39,  5.11it/s]
 37%|███▋      | 3669/10000 [25:08<20:49,  5.07it/s]
 37%|███▋      | 3670/10000 [25:08<20:58,  5.03it/s]
                                                    
{'loss': 0.1052, 'grad_norm': 0.8769537806510925, 'learning_rate': 7.496658046731096e-05}

 37%|███▋      | 3670/10000 [25:08<20:58,  5.03it/s]
 37%|███▋      | 3671/10000 [25:08<21:27,  4.92it/s]
 37%|███▋      | 3672/10000 [25:08<22:05,  4.77it/s]
 37%|███▋      | 3673/10000 [25:09<22:05,  4.77it/s]
 37%|███▋      | 3674/10000 [25:09<21:38,  4.87it/s]
 37%|███▋      | 3675/10000 [25:09<21:07,  4.99it/s]
 37%|███▋      | 3676/10000 [25:09<20:28,  5.15it/s]
 37%|███▋      | 3677/10000 [25:09<19:59,  5.27it/s]
 37%|███▋      | 3678/10000 [25:10<20:21,  5.18it/s]
 37%|███▋      | 3679/10000 [25:10<20:45,  5.07it/s]
 37%|███▋      | 3680/10000 [25:10<20:54,  5.04it/s]
                                                    
{'loss': 0.1116, 'grad_norm': 1.1584025621414185, 'learning_rate': 7.482318577668578e-05}

 37%|███▋      | 3680/10000 [25:10<20:54,  5.04it/s]
 37%|███▋      | 3681/10000 [25:10<21:23,  4.92it/s]
 37%|███▋      | 3682/10000 [25:10<21:14,  4.96it/s]
 37%|███▋      | 3683/10000 [25:11<21:11,  4.97it/s]
 37%|███▋      | 3684/10000 [25:11<20:57,  5.02it/s]
 37%|███▋      | 3685/10000 [25:11<20:23,  5.16it/s]
 37%|███▋      | 3686/10000 [25:11<19:47,  5.32it/s]
 37%|███▋      | 3687/10000 [25:11<19:33,  5.38it/s]
 37%|███▋      | 3688/10000 [25:12<20:06,  5.23it/s]
 37%|███▋      | 3689/10000 [25:12<21:03,  4.99it/s]
 37%|███▋      | 3690/10000 [25:12<21:42,  4.85it/s]
                                                    
{'loss': 0.1053, 'grad_norm': 0.9424688816070557, 'learning_rate': 7.467951962367796e-05}

 37%|███▋      | 3690/10000 [25:12<21:42,  4.85it/s]
 37%|███▋      | 3691/10000 [25:12<22:18,  4.71it/s]
 37%|███▋      | 3692/10000 [25:12<21:53,  4.80it/s]
 37%|███▋      | 3693/10000 [25:13<21:13,  4.95it/s]
 37%|███▋      | 3694/10000 [25:13<20:27,  5.14it/s]
 37%|███▋      | 3695/10000 [25:13<20:16,  5.18it/s]
 37%|███▋      | 3696/10000 [25:13<20:23,  5.15it/s]
 37%|███▋      | 3697/10000 [25:13<20:44,  5.06it/s]
 37%|███▋      | 3698/10000 [25:14<21:19,  4.92it/s]
 37%|███▋      | 3699/10000 [25:14<21:35,  4.87it/s]
 37%|███▋      | 3700/10000 [25:14<21:55,  4.79it/s]
                                                    
{'loss': 0.1226, 'grad_norm': 0.8237622976303101, 'learning_rate': 7.453558357939755e-05}

 37%|███▋      | 3700/10000 [25:14<21:55,  4.79it/s]
 37%|███▋      | 3701/10000 [25:14<21:31,  4.88it/s]
 37%|███▋      | 3702/10000 [25:14<20:49,  5.04it/s]
 37%|███▋      | 3703/10000 [25:15<20:37,  5.09it/s]
 37%|███▋      | 3704/10000 [25:15<20:46,  5.05it/s]
 37%|███▋      | 3705/10000 [25:15<21:19,  4.92it/s]
 37%|███▋      | 3706/10000 [25:15<22:05,  4.75it/s]
 37%|███▋      | 3707/10000 [25:15<21:45,  4.82it/s]
 37%|███▋      | 3708/10000 [25:16<22:07,  4.74it/s]
 37%|███▋      | 3709/10000 [25:16<21:22,  4.91it/s]
 37%|███▋      | 3710/10000 [25:16<20:30,  5.11it/s]
                                                    
{'loss': 0.1013, 'grad_norm': 0.9490163326263428, 'learning_rate': 7.439137921790606e-05}

 37%|███▋      | 3710/10000 [25:16<20:30,  5.11it/s]
 37%|███▋      | 3711/10000 [25:16<20:31,  5.11it/s]
 37%|███▋      | 3712/10000 [25:16<20:32,  5.10it/s]
 37%|███▋      | 3713/10000 [25:17<21:01,  4.99it/s]
 37%|███▋      | 3714/10000 [25:17<21:10,  4.95it/s]
 37%|███▋      | 3715/10000 [25:17<21:25,  4.89it/s]
 37%|███▋      | 3716/10000 [25:17<21:35,  4.85it/s]
 37%|███▋      | 3717/10000 [25:17<20:43,  5.05it/s]
 37%|███▋      | 3718/10000 [25:18<19:55,  5.25it/s]
 37%|███▋      | 3719/10000 [25:18<19:36,  5.34it/s]
 37%|███▋      | 3720/10000 [25:18<19:58,  5.24it/s]
                                                    
{'loss': 0.141, 'grad_norm': 0.8553034067153931, 'learning_rate': 7.42469081161993e-05}

 37%|███▋      | 3720/10000 [25:18<19:58,  5.24it/s]
 37%|███▋      | 3721/10000 [25:18<20:54,  5.00it/s]
 37%|███▋      | 3722/10000 [25:18<21:07,  4.95it/s]
 37%|███▋      | 3723/10000 [25:19<21:24,  4.89it/s]
 37%|███▋      | 3724/10000 [25:19<21:11,  4.94it/s]
 37%|███▋      | 3725/10000 [25:19<20:50,  5.02it/s]
 37%|███▋      | 3726/10000 [25:19<20:02,  5.22it/s]
 37%|███▋      | 3727/10000 [25:19<19:32,  5.35it/s]
 37%|███▋      | 3728/10000 [25:20<19:48,  5.28it/s]
 37%|███▋      | 3729/10000 [25:20<20:07,  5.19it/s]
 37%|███▋      | 3730/10000 [25:20<20:09,  5.18it/s]
                                                    
{'loss': 0.1057, 'grad_norm': 0.8201740384101868, 'learning_rate': 7.410217185419006e-05}

 37%|███▋      | 3730/10000 [25:20<20:09,  5.18it/s]
 37%|███▋      | 3731/10000 [25:20<20:33,  5.08it/s]
 37%|███▋      | 3732/10000 [25:20<19:47,  5.28it/s]
 37%|███▋      | 3733/10000 [25:21<19:13,  5.43it/s]
 37%|███▋      | 3734/10000 [25:21<18:45,  5.57it/s]
 37%|███▋      | 3735/10000 [25:21<18:29,  5.64it/s]
 37%|███▋      | 3736/10000 [25:21<18:37,  5.60it/s]
 37%|███▋      | 3737/10000 [25:21<18:36,  5.61it/s]
 37%|███▋      | 3738/10000 [25:21<18:19,  5.69it/s]
 37%|███▋      | 3739/10000 [25:22<18:06,  5.76it/s]
 37%|███▋      | 3740/10000 [25:22<17:57,  5.81it/s]
                                                    
{'loss': 0.1049, 'grad_norm': 0.7229145169258118, 'learning_rate': 7.395717201469095e-05}

 37%|███▋      | 3740/10000 [25:22<17:57,  5.81it/s]
 37%|███▋      | 3741/10000 [25:22<18:04,  5.77it/s]
 37%|███▋      | 3742/10000 [25:22<18:21,  5.68it/s]
 37%|███▋      | 3743/10000 [25:22<19:32,  5.34it/s]
 37%|███▋      | 3744/10000 [25:23<20:09,  5.17it/s]
 37%|███▋      | 3745/10000 [25:23<20:18,  5.14it/s]Rank 0, Worker 1: Wait for shard 10 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 37%|███▋      | 3746/10000 [25:23<19:41,  5.29it/s]
 37%|███▋      | 3747/10000 [25:23<19:12,  5.42it/s]
 37%|███▋      | 3748/10000 [25:23<18:48,  5.54it/s]
 37%|███▋      | 3749/10000 [25:23<18:30,  5.63it/s]
 38%|███▊      | 3750/10000 [25:24<18:18,  5.69it/s]Rank 0, Worker 1: Caching shard...Rank 0, Worker 0: Caching shard...Rank 0, Worker 2: Caching shard...


Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 14.50 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 15.84 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 15.95 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 16.20 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 16.37 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 16.64 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 3750: eval/loss = 0.1172 (34s)

                                                    
{'loss': 0.133, 'grad_norm': 0.7403004169464111, 'learning_rate': 7.381191018339696e-05}

 38%|███▊      | 3750/10000 [25:57<18:18,  5.69it/s]
 38%|███▊      | 3751/10000 [25:57<17:49:52, 10.27s/it]
 38%|███▊      | 3752/10000 [25:58<12:34:26,  7.25s/it]
 38%|███▊      | 3753/10000 [25:58<8:54:26,  5.13s/it] 
 38%|███▊      | 3754/10000 [25:58<6:20:27,  3.65s/it]
 38%|███▊      | 3755/10000 [25:58<4:32:47,  2.62s/it]
 38%|███▊      | 3756/10000 [25:58<3:17:07,  1.89s/it]
 38%|███▊      | 3757/10000 [25:59<2:23:50,  1.38s/it]
 38%|███▊      | 3758/10000 [25:59<1:46:28,  1.02s/it]
 38%|███▊      | 3759/10000 [25:59<1:20:11,  1.30it/s]
 38%|███▊      | 3760/10000 [25:59<1:01:39,  1.69it/s]
                                                      
{'loss': 0.1424, 'grad_norm': 0.9609218239784241, 'learning_rate': 7.36663879488682e-05}

 38%|███▊      | 3760/10000 [25:59<1:01:39,  1.69it/s]
 38%|███▊      | 3761/10000 [25:59<49:20,  2.11it/s]  
 38%|███▊      | 3762/10000 [26:00<40:30,  2.57it/s]
 38%|███▊      | 3763/10000 [26:00<34:25,  3.02it/s]
 38%|███▊      | 3764/10000 [26:00<30:11,  3.44it/s]
 38%|███▊      | 3765/10000 [26:00<27:16,  3.81it/s]
 38%|███▊      | 3766/10000 [26:00<25:22,  4.09it/s]
 38%|███▊      | 3767/10000 [26:01<23:50,  4.36it/s]
 38%|███▊      | 3768/10000 [26:01<23:00,  4.51it/s]
 38%|███▊      | 3769/10000 [26:01<22:11,  4.68it/s]
 38%|███▊      | 3770/10000 [26:01<21:18,  4.87it/s]
                                                    
{'loss': 0.1077, 'grad_norm': 0.8832553625106812, 'learning_rate': 7.352060690251254e-05}

 38%|███▊      | 3770/10000 [26:01<21:18,  4.87it/s]
 38%|███▊      | 3771/10000 [26:01<21:27,  4.84it/s]
 38%|███▊      | 3772/10000 [26:02<20:58,  4.95it/s]
 38%|███▊      | 3773/10000 [26:02<20:23,  5.09it/s]
 38%|███▊      | 3774/10000 [26:02<20:16,  5.12it/s]
 38%|███▊      | 3775/10000 [26:02<20:16,  5.12it/s]
 38%|███▊      | 3776/10000 [26:02<20:29,  5.06it/s]
 38%|███▊      | 3777/10000 [26:02<21:01,  4.93it/s]
 38%|███▊      | 3778/10000 [26:03<20:49,  4.98it/s]
 38%|███▊      | 3779/10000 [26:03<20:45,  4.99it/s]
 38%|███▊      | 3780/10000 [26:03<20:01,  5.18it/s]
                                                    
{'loss': 0.1112, 'grad_norm': 0.8751207590103149, 'learning_rate': 7.337456863856811e-05}

 38%|███▊      | 3780/10000 [26:03<20:01,  5.18it/s]
 38%|███▊      | 3781/10000 [26:03<19:54,  5.21it/s]
 38%|███▊      | 3782/10000 [26:03<19:42,  5.26it/s]
 38%|███▊      | 3783/10000 [26:04<19:45,  5.24it/s]
 38%|███▊      | 3784/10000 [26:04<19:59,  5.18it/s]
 38%|███▊      | 3785/10000 [26:04<20:18,  5.10it/s]
 38%|███▊      | 3786/10000 [26:04<20:22,  5.08it/s]
 38%|███▊      | 3787/10000 [26:04<20:07,  5.15it/s]
 38%|███▊      | 3788/10000 [26:05<19:49,  5.22it/s]
 38%|███▊      | 3789/10000 [26:05<19:27,  5.32it/s]
 38%|███▊      | 3790/10000 [26:05<19:10,  5.40it/s]
                                                    
{'loss': 0.1392, 'grad_norm': 0.7030627727508545, 'learning_rate': 7.3228274754086e-05}

 38%|███▊      | 3790/10000 [26:05<19:10,  5.40it/s]
 38%|███▊      | 3791/10000 [26:05<19:28,  5.32it/s]
 38%|███▊      | 3792/10000 [26:05<19:32,  5.29it/s]
 38%|███▊      | 3793/10000 [26:06<19:30,  5.30it/s]
 38%|███▊      | 3794/10000 [26:06<19:33,  5.29it/s]
 38%|███▊      | 3795/10000 [26:06<19:11,  5.39it/s]Rank 0, Worker 3: Wait for shard 32 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 38%|███▊      | 3796/10000 [26:06<19:21,  5.34it/s]
 38%|███▊      | 3797/10000 [26:06<19:15,  5.37it/s]
 38%|███▊      | 3798/10000 [26:06<19:14,  5.37it/s]
 38%|███▊      | 3799/10000 [26:07<19:19,  5.35it/s]
 38%|███▊      | 3800/10000 [26:07<19:16,  5.36it/s]
                                                    
{'loss': 0.1034, 'grad_norm': 0.8281252384185791, 'learning_rate': 7.308172684891267e-05}

 38%|███▊      | 3800/10000 [26:07<19:16,  5.36it/s]
 38%|███▊      | 3801/10000 [26:07<19:18,  5.35it/s]
 38%|███▊      | 3802/10000 [26:07<19:10,  5.39it/s]
 38%|███▊      | 3803/10000 [26:07<19:01,  5.43it/s]
 38%|███▊      | 3804/10000 [26:08<19:14,  5.37it/s]
 38%|███▊      | 3805/10000 [26:08<19:24,  5.32it/s]
 38%|███▊      | 3806/10000 [26:08<19:18,  5.35it/s]
 38%|███▊      | 3807/10000 [26:08<19:23,  5.32it/s]
 38%|███▊      | 3808/10000 [26:08<19:04,  5.41it/s]
 38%|███▊      | 3809/10000 [26:09<19:47,  5.21it/s]
 38%|███▊      | 3810/10000 [26:09<21:00,  4.91it/s]
                                                    
{'loss': 0.1076, 'grad_norm': 0.8161336779594421, 'learning_rate': 7.293492652567255e-05}

 38%|███▊      | 3810/10000 [26:09<21:00,  4.91it/s]
 38%|███▊      | 3811/10000 [26:09<20:57,  4.92it/s]
 38%|███▊      | 3812/10000 [26:09<20:45,  4.97it/s]
 38%|███▊      | 3813/10000 [26:09<19:58,  5.16it/s]
 38%|███▊      | 3814/10000 [26:10<19:26,  5.30it/s]
 38%|███▊      | 3815/10000 [26:10<19:07,  5.39it/s]
 38%|███▊      | 3816/10000 [26:10<18:52,  5.46it/s]
 38%|███▊      | 3817/10000 [26:10<18:41,  5.51it/s]
 38%|███▊      | 3818/10000 [26:10<18:38,  5.53it/s]
 38%|███▊      | 3819/10000 [26:10<18:30,  5.57it/s]
 38%|███▊      | 3820/10000 [26:11<18:35,  5.54it/s]
                                                    
{'loss': 0.1255, 'grad_norm': 0.8957645297050476, 'learning_rate': 7.278787538975043e-05}

 38%|███▊      | 3820/10000 [26:11<18:35,  5.54it/s]
 38%|███▊      | 3821/10000 [26:11<19:28,  5.29it/s]
 38%|███▊      | 3822/10000 [26:11<19:35,  5.25it/s]
 38%|███▊      | 3823/10000 [26:11<19:47,  5.20it/s]
 38%|███▊      | 3824/10000 [26:11<20:11,  5.10it/s]
 38%|███▊      | 3825/10000 [26:12<20:31,  5.01it/s]
 38%|███▊      | 3826/10000 [26:12<20:06,  5.12it/s]
 38%|███▊      | 3827/10000 [26:12<20:58,  4.90it/s]
 38%|███▊      | 3828/10000 [26:12<21:24,  4.80it/s]
 38%|███▊      | 3829/10000 [26:12<21:30,  4.78it/s]
 38%|███▊      | 3830/10000 [26:13<21:01,  4.89it/s]
                                                    
{'loss': 0.1076, 'grad_norm': 0.8216802477836609, 'learning_rate': 7.2640575049274e-05}

 38%|███▊      | 3830/10000 [26:13<21:01,  4.89it/s]
 38%|███▊      | 3831/10000 [26:13<20:35,  4.99it/s]
 38%|███▊      | 3832/10000 [26:13<20:13,  5.08it/s]
 38%|███▊      | 3833/10000 [26:13<20:06,  5.11it/s]
 38%|███▊      | 3834/10000 [26:13<20:12,  5.09it/s]
 38%|███▊      | 3835/10000 [26:14<20:14,  5.08it/s]
 38%|███▊      | 3836/10000 [26:14<20:18,  5.06it/s]
 38%|███▊      | 3837/10000 [26:14<20:19,  5.05it/s]
 38%|███▊      | 3838/10000 [26:14<20:16,  5.07it/s]
 38%|███▊      | 3839/10000 [26:14<19:46,  5.19it/s]Rank 0, Worker 5: Wait for shard 4 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 38%|███▊      | 3840/10000 [26:15<19:33,  5.25it/s]
                                                    
{'loss': 0.1085, 'grad_norm': 0.8624475598335266, 'learning_rate': 7.249302711509616e-05}

 38%|███▊      | 3840/10000 [26:15<19:33,  5.25it/s]
 38%|███▊      | 3841/10000 [26:15<19:47,  5.19it/s]
 38%|███▊      | 3842/10000 [26:15<19:59,  5.13it/s]
 38%|███▊      | 3843/10000 [26:15<20:08,  5.09it/s]
 38%|███▊      | 3844/10000 [26:15<20:23,  5.03it/s]
 38%|███▊      | 3845/10000 [26:16<20:25,  5.02it/s]
 38%|███▊      | 3846/10000 [26:16<20:22,  5.03it/s]
 38%|███▊      | 3847/10000 [26:16<20:01,  5.12it/s]
 38%|███▊      | 3848/10000 [26:16<20:00,  5.12it/s]
 38%|███▊      | 3849/10000 [26:16<19:59,  5.13it/s]
 38%|███▊      | 3850/10000 [26:17<20:13,  5.07it/s]
                                                    
{'loss': 0.1014, 'grad_norm': 0.8596972227096558, 'learning_rate': 7.23452332007775e-05}

 38%|███▊      | 3850/10000 [26:17<20:13,  5.07it/s]
 39%|███▊      | 3851/10000 [26:17<20:39,  4.96it/s]
 39%|███▊      | 3852/10000 [26:17<20:37,  4.97it/s]
 39%|███▊      | 3853/10000 [26:17<21:00,  4.88it/s]
 39%|███▊      | 3854/10000 [26:17<20:53,  4.90it/s]
 39%|███▊      | 3855/10000 [26:18<20:31,  4.99it/s]
 39%|███▊      | 3856/10000 [26:18<20:30,  4.99it/s]
 39%|███▊      | 3857/10000 [26:18<20:36,  4.97it/s]
 39%|███▊      | 3858/10000 [26:18<20:37,  4.96it/s]
 39%|███▊      | 3859/10000 [26:18<20:25,  5.01it/s]
 39%|███▊      | 3860/10000 [26:19<20:18,  5.04it/s]
                                                    
{'loss': 0.115, 'grad_norm': 0.8467769622802734, 'learning_rate': 7.219719492256858e-05}

 39%|███▊      | 3860/10000 [26:19<20:18,  5.04it/s]
 39%|███▊      | 3861/10000 [26:19<20:59,  4.87it/s]
 39%|███▊      | 3862/10000 [26:19<21:10,  4.83it/s]
 39%|███▊      | 3863/10000 [26:19<21:07,  4.84it/s]
 39%|███▊      | 3864/10000 [26:19<21:04,  4.85it/s]
 39%|███▊      | 3865/10000 [26:20<21:02,  4.86it/s]
 39%|███▊      | 3866/10000 [26:20<20:46,  4.92it/s]
 39%|███▊      | 3867/10000 [26:20<20:35,  4.96it/s]
 39%|███▊      | 3868/10000 [26:20<20:20,  5.02it/s]
 39%|███▊      | 3869/10000 [26:20<20:06,  5.08it/s]
 39%|███▊      | 3870/10000 [26:21<20:14,  5.05it/s]
                                                    
{'loss': 0.0906, 'grad_norm': 0.8898311853408813, 'learning_rate': 7.20489138993923e-05}

 39%|███▊      | 3870/10000 [26:21<20:14,  5.05it/s]
 39%|███▊      | 3871/10000 [26:21<21:17,  4.80it/s]
 39%|███▊      | 3872/10000 [26:21<21:28,  4.76it/s]
 39%|███▊      | 3873/10000 [26:21<21:13,  4.81it/s]
 39%|███▊      | 3874/10000 [26:21<20:47,  4.91it/s]
 39%|███▉      | 3875/10000 [26:22<20:12,  5.05it/s]
 39%|███▉      | 3876/10000 [26:22<20:02,  5.09it/s]
 39%|███▉      | 3877/10000 [26:22<19:57,  5.11it/s]
 39%|███▉      | 3878/10000 [26:22<19:56,  5.12it/s]
 39%|███▉      | 3879/10000 [26:22<20:14,  5.04it/s]
 39%|███▉      | 3880/10000 [26:23<20:08,  5.06it/s]
                                                    
{'loss': 0.137, 'grad_norm': 1.0945919752120972, 'learning_rate': 7.190039175282614e-05}

 39%|███▉      | 3880/10000 [26:23<20:08,  5.06it/s]
 39%|███▉      | 3881/10000 [26:23<21:25,  4.76it/s]
 39%|███▉      | 3882/10000 [26:23<20:37,  4.94it/s]
 39%|███▉      | 3883/10000 [26:23<20:01,  5.09it/s]
 39%|███▉      | 3884/10000 [26:23<19:48,  5.15it/s]
 39%|███▉      | 3885/10000 [26:24<19:47,  5.15it/s]
 39%|███▉      | 3886/10000 [26:24<19:19,  5.27it/s]
 39%|███▉      | 3887/10000 [26:24<19:38,  5.19it/s]
 39%|███▉      | 3888/10000 [26:24<20:05,  5.07it/s]
 39%|███▉      | 3889/10000 [26:24<20:27,  4.98it/s]
 39%|███▉      | 3890/10000 [26:25<20:46,  4.90it/s]
                                                    
{'loss': 0.1141, 'grad_norm': 0.7952854633331299, 'learning_rate': 7.175163010708455e-05}

 39%|███▉      | 3890/10000 [26:25<20:46,  4.90it/s]
 39%|███▉      | 3891/10000 [26:25<21:13,  4.80it/s]
 39%|███▉      | 3892/10000 [26:25<20:57,  4.86it/s]
 39%|███▉      | 3893/10000 [26:25<20:41,  4.92it/s]
 39%|███▉      | 3894/10000 [26:25<20:27,  4.98it/s]
 39%|███▉      | 3895/10000 [26:26<20:03,  5.07it/s]
 39%|███▉      | 3896/10000 [26:26<20:08,  5.05it/s]
 39%|███▉      | 3897/10000 [26:26<20:09,  5.04it/s]
 39%|███▉      | 3898/10000 [26:26<20:33,  4.95it/s]
 39%|███▉      | 3899/10000 [26:26<20:56,  4.86it/s]
 39%|███▉      | 3900/10000 [26:27<20:50,  4.88it/s]
                                                    
{'loss': 0.1011, 'grad_norm': 0.9095091819763184, 'learning_rate': 7.1602630589001e-05}

 39%|███▉      | 3900/10000 [26:27<20:50,  4.88it/s]
 39%|███▉      | 3901/10000 [26:27<21:10,  4.80it/s]
 39%|███▉      | 3902/10000 [26:27<21:06,  4.81it/s]Rank 0, Worker 2: Wait for shard 24 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 39%|███▉      | 3903/10000 [26:27<20:56,  4.85it/s]
 39%|███▉      | 3904/10000 [26:27<20:57,  4.85it/s]
 39%|███▉      | 3905/10000 [26:28<21:04,  4.82it/s]
 39%|███▉      | 3906/10000 [26:28<21:06,  4.81it/s]
 39%|███▉      | 3907/10000 [26:28<21:24,  4.75it/s]
 39%|███▉      | 3908/10000 [26:28<20:51,  4.87it/s]
 39%|███▉      | 3909/10000 [26:28<20:40,  4.91it/s]
 39%|███▉      | 3910/10000 [26:29<20:31,  4.94it/s]
                                                    
{'loss': 0.103, 'grad_norm': 0.8677197098731995, 'learning_rate': 7.14533948280104e-05}

 39%|███▉      | 3910/10000 [26:29<20:31,  4.94it/s]
 39%|███▉      | 3911/10000 [26:29<20:29,  4.95it/s]
 39%|███▉      | 3912/10000 [26:29<20:20,  4.99it/s]Rank 0, Worker 0: Wait for shard 37 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 39%|███▉      | 3913/10000 [26:29<20:27,  4.96it/s]
 39%|███▉      | 3914/10000 [26:29<20:19,  4.99it/s]
 39%|███▉      | 3915/10000 [26:30<20:25,  4.96it/s]
 39%|███▉      | 3916/10000 [26:30<20:26,  4.96it/s]
 39%|███▉      | 3917/10000 [26:30<20:13,  5.01it/s]
 39%|███▉      | 3918/10000 [26:30<20:07,  5.04it/s]
 39%|███▉      | 3919/10000 [26:30<19:52,  5.10it/s]
 39%|███▉      | 3920/10000 [26:31<19:27,  5.21it/s]
                                                    
{'loss': 0.1048, 'grad_norm': 0.8985068202018738, 'learning_rate': 7.130392445613109e-05}

 39%|███▉      | 3920/10000 [26:31<19:27,  5.21it/s]
 39%|███▉      | 3921/10000 [26:31<19:57,  5.07it/s]
 39%|███▉      | 3922/10000 [26:31<19:55,  5.08it/s]
 39%|███▉      | 3923/10000 [26:31<19:45,  5.13it/s]
 39%|███▉      | 3924/10000 [26:31<19:40,  5.15it/s]
 39%|███▉      | 3925/10000 [26:32<19:34,  5.17it/s]
 39%|███▉      | 3926/10000 [26:32<19:24,  5.22it/s]
 39%|███▉      | 3927/10000 [26:32<19:34,  5.17it/s]
 39%|███▉      | 3928/10000 [26:32<19:39,  5.15it/s]
 39%|███▉      | 3929/10000 [26:32<19:35,  5.16it/s]
 39%|███▉      | 3930/10000 [26:33<19:28,  5.19it/s]
                                                    
{'loss': 0.1139, 'grad_norm': 0.8469200730323792, 'learning_rate': 7.115422110794711e-05}

 39%|███▉      | 3930/10000 [26:33<19:28,  5.19it/s]
 39%|███▉      | 3931/10000 [26:33<19:38,  5.15it/s]
 39%|███▉      | 3932/10000 [26:33<19:15,  5.25it/s]
 39%|███▉      | 3933/10000 [26:33<19:06,  5.29it/s]
 39%|███▉      | 3934/10000 [26:33<18:44,  5.40it/s]
 39%|███▉      | 3935/10000 [26:34<18:26,  5.48it/s]
 39%|███▉      | 3936/10000 [26:34<18:57,  5.33it/s]
 39%|███▉      | 3937/10000 [26:34<19:31,  5.17it/s]
 39%|███▉      | 3938/10000 [26:34<19:50,  5.09it/s]
 39%|███▉      | 3939/10000 [26:34<19:54,  5.08it/s]
 39%|███▉      | 3940/10000 [26:35<19:46,  5.11it/s]
                                                    
{'loss': 0.1122, 'grad_norm': 0.7152714729309082, 'learning_rate': 7.100428642059033e-05}

 39%|███▉      | 3940/10000 [26:35<19:46,  5.11it/s]
 39%|███▉      | 3941/10000 [26:35<19:41,  5.13it/s]
 39%|███▉      | 3942/10000 [26:35<19:25,  5.20it/s]
 39%|███▉      | 3943/10000 [26:35<19:02,  5.30it/s]
 39%|███▉      | 3944/10000 [26:35<19:08,  5.27it/s]
 39%|███▉      | 3945/10000 [26:35<19:59,  5.05it/s]
 39%|███▉      | 3946/10000 [26:36<20:02,  5.04it/s]
 39%|███▉      | 3947/10000 [26:36<19:56,  5.06it/s]
 39%|███▉      | 3948/10000 [26:36<19:45,  5.11it/s]
 39%|███▉      | 3949/10000 [26:36<19:32,  5.16it/s]
 40%|███▉      | 3950/10000 [26:36<19:39,  5.13it/s]
                                                    
{'loss': 0.1041, 'grad_norm': 0.9668436646461487, 'learning_rate': 7.08541220337224e-05}

 40%|███▉      | 3950/10000 [26:36<19:39,  5.13it/s]
 40%|███▉      | 3951/10000 [26:37<19:51,  5.08it/s]
 40%|███▉      | 3952/10000 [26:37<19:49,  5.08it/s]
 40%|███▉      | 3953/10000 [26:37<20:00,  5.04it/s]
 40%|███▉      | 3954/10000 [26:37<21:00,  4.80it/s]
 40%|███▉      | 3955/10000 [26:37<20:07,  5.00it/s]
 40%|███▉      | 3956/10000 [26:38<19:59,  5.04it/s]
 40%|███▉      | 3957/10000 [26:38<19:50,  5.08it/s]
 40%|███▉      | 3958/10000 [26:38<19:37,  5.13it/s]
 40%|███▉      | 3959/10000 [26:38<20:01,  5.03it/s]
 40%|███▉      | 3960/10000 [26:38<19:51,  5.07it/s]
                                                    
{'loss': 0.106, 'grad_norm': 0.9783576130867004, 'learning_rate': 7.070372958951706e-05}

 40%|███▉      | 3960/10000 [26:38<19:51,  5.07it/s]
 40%|███▉      | 3961/10000 [26:39<19:55,  5.05it/s]
 40%|███▉      | 3962/10000 [26:39<19:52,  5.06it/s]
 40%|███▉      | 3963/10000 [26:39<19:39,  5.12it/s]
 40%|███▉      | 3964/10000 [26:39<19:43,  5.10it/s]
 40%|███▉      | 3965/10000 [26:39<20:31,  4.90it/s]
 40%|███▉      | 3966/10000 [26:40<20:26,  4.92it/s]
 40%|███▉      | 3967/10000 [26:40<21:09,  4.75it/s]
 40%|███▉      | 3968/10000 [26:40<21:30,  4.68it/s]
 40%|███▉      | 3969/10000 [26:40<20:41,  4.86it/s]
 40%|███▉      | 3970/10000 [26:40<20:06,  5.00it/s]
                                                    
{'loss': 0.1044, 'grad_norm': 0.9123092889785767, 'learning_rate': 7.055311073264194e-05}

 40%|███▉      | 3970/10000 [26:41<20:06,  5.00it/s]
 40%|███▉      | 3971/10000 [26:41<20:10,  4.98it/s]
 40%|███▉      | 3972/10000 [26:41<19:47,  5.08it/s]
 40%|███▉      | 3973/10000 [26:41<20:32,  4.89it/s]
 40%|███▉      | 3974/10000 [26:41<21:27,  4.68it/s]
 40%|███▉      | 3975/10000 [26:42<21:51,  4.59it/s]
 40%|███▉      | 3976/10000 [26:42<22:26,  4.47it/s]
 40%|███▉      | 3977/10000 [26:42<23:07,  4.34it/s]
 40%|███▉      | 3978/10000 [26:42<23:21,  4.30it/s]
 40%|███▉      | 3979/10000 [26:42<22:52,  4.39it/s]
 40%|███▉      | 3980/10000 [26:43<22:01,  4.55it/s]
                                                    
{'loss': 0.124, 'grad_norm': 0.7978675961494446, 'learning_rate': 7.040226711024077e-05}

 40%|███▉      | 3980/10000 [26:43<22:01,  4.55it/s]
 40%|███▉      | 3981/10000 [26:43<21:36,  4.64it/s]
 40%|███▉      | 3982/10000 [26:43<20:34,  4.88it/s]
 40%|███▉      | 3983/10000 [26:43<20:21,  4.93it/s]
 40%|███▉      | 3984/10000 [26:44<21:29,  4.67it/s]
 40%|███▉      | 3985/10000 [26:44<21:58,  4.56it/s]
 40%|███▉      | 3986/10000 [26:44<22:36,  4.43it/s]
 40%|███▉      | 3987/10000 [26:44<23:02,  4.35it/s]
 40%|███▉      | 3988/10000 [26:44<22:21,  4.48it/s]
 40%|███▉      | 3989/10000 [26:45<21:55,  4.57it/s]
 40%|███▉      | 3990/10000 [26:45<21:28,  4.66it/s]
                                                    
{'loss': 0.1107, 'grad_norm': 0.8108384013175964, 'learning_rate': 7.02512003719152e-05}

 40%|███▉      | 3990/10000 [26:45<21:28,  4.66it/s]
 40%|███▉      | 3991/10000 [26:45<20:51,  4.80it/s]
 40%|███▉      | 3992/10000 [26:45<20:11,  4.96it/s]
 40%|███▉      | 3993/10000 [26:45<20:36,  4.86it/s]
 40%|███▉      | 3994/10000 [26:46<21:45,  4.60it/s]
 40%|███▉      | 3995/10000 [26:46<22:45,  4.40it/s]
 40%|███▉      | 3996/10000 [26:46<23:13,  4.31it/s]
 40%|███▉      | 3997/10000 [26:46<23:04,  4.34it/s]
 40%|███▉      | 3998/10000 [26:47<22:25,  4.46it/s]
 40%|███▉      | 3999/10000 [26:47<21:12,  4.72it/s]
 40%|████      | 4000/10000 [26:47<20:22,  4.91it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...Rank 0, Worker 2: Caching shard...

Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 18.03 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.49 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.50 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.73 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.79 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.89 seconds
Rank 0, Worker 0: Caching shard...
[held-out-eval] step 4000: eval/loss = 0.1227 (39s)

                                                    
{'loss': 0.1091, 'grad_norm': 0.9918605089187622, 'learning_rate': 7.00999121697069e-05}

 40%|████      | 4000/10000 [27:26<20:22,  4.91it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-4000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-4000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-4000/wandb_config.json

 40%|████      | 4001/10000 [27:53<33:27:30, 20.08s/it]
 40%|████      | 4002/10000 [27:54<23:30:43, 14.11s/it]
 40%|████      | 4003/10000 [27:54<16:32:34,  9.93s/it]
 40%|████      | 4004/10000 [27:54<11:40:02,  7.01s/it]
 40%|████      | 4005/10000 [27:54<8:15:16,  4.96s/it] 
 40%|████      | 4006/10000 [27:54<5:52:27,  3.53s/it]
 40%|████      | 4007/10000 [27:55<4:13:04,  2.53s/it]
 40%|████      | 4008/10000 [27:55<3:03:29,  1.84s/it]
 40%|████      | 4009/10000 [27:55<2:14:22,  1.35s/it]
 40%|████      | 4010/10000 [27:55<1:41:04,  1.01s/it]
                                                      
{'loss': 0.1086, 'grad_norm': 0.9199183583259583, 'learning_rate': 6.99484041580794e-05}

 40%|████      | 4010/10000 [27:55<1:41:04,  1.01s/it]
 40%|████      | 4011/10000 [27:55<1:16:12,  1.31it/s]
 40%|████      | 4012/10000 [27:56<58:41,  1.70it/s]  
 40%|████      | 4013/10000 [27:56<46:31,  2.14it/s]
 40%|████      | 4014/10000 [27:56<38:43,  2.58it/s]
 40%|████      | 4015/10000 [27:56<33:34,  2.97it/s]
 40%|████      | 4016/10000 [27:56<29:48,  3.35it/s]
 40%|████      | 4017/10000 [27:57<26:44,  3.73it/s]
 40%|████      | 4018/10000 [27:57<24:04,  4.14it/s]
 40%|████      | 4019/10000 [27:57<22:08,  4.50it/s]
 40%|████      | 4020/10000 [27:57<20:45,  4.80it/s]
                                                    
{'loss': 0.1223, 'grad_norm': 0.7610146999359131, 'learning_rate': 6.979667799390004e-05}

 40%|████      | 4020/10000 [27:57<20:45,  4.80it/s]
 40%|████      | 4021/10000 [27:57<20:09,  4.94it/s]
 40%|████      | 4022/10000 [27:57<19:32,  5.10it/s]
 40%|████      | 4023/10000 [27:58<19:16,  5.17it/s]
 40%|████      | 4024/10000 [27:58<19:39,  5.07it/s]
 40%|████      | 4025/10000 [27:58<19:42,  5.05it/s]
 40%|████      | 4026/10000 [27:58<19:02,  5.23it/s]
 40%|████      | 4027/10000 [27:58<18:38,  5.34it/s]
 40%|████      | 4028/10000 [27:59<18:11,  5.47it/s]
 40%|████      | 4029/10000 [27:59<17:52,  5.57it/s]
 40%|████      | 4030/10000 [27:59<17:49,  5.58it/s]
                                                    
{'loss': 0.1169, 'grad_norm': 1.0439714193344116, 'learning_rate': 6.964473533642185e-05}

 40%|████      | 4030/10000 [27:59<17:49,  5.58it/s]
 40%|████      | 4031/10000 [27:59<18:24,  5.41it/s]
 40%|████      | 4032/10000 [27:59<18:25,  5.40it/s]
 40%|████      | 4033/10000 [28:00<18:15,  5.44it/s]
 40%|████      | 4034/10000 [28:00<18:09,  5.48it/s]
 40%|████      | 4035/10000 [28:00<17:49,  5.58it/s]
 40%|████      | 4036/10000 [28:00<17:43,  5.61it/s]
 40%|████      | 4037/10000 [28:00<17:43,  5.61it/s]
 40%|████      | 4038/10000 [28:00<17:40,  5.62it/s]
 40%|████      | 4039/10000 [28:01<17:52,  5.56it/s]
 40%|████      | 4040/10000 [28:01<18:05,  5.49it/s]
                                                    
{'loss': 0.1203, 'grad_norm': 0.8141038417816162, 'learning_rate': 6.949257784726539e-05}

 40%|████      | 4040/10000 [28:01<18:05,  5.49it/s]
 40%|████      | 4041/10000 [28:01<18:12,  5.45it/s]
 40%|████      | 4042/10000 [28:01<18:02,  5.50it/s]
 40%|████      | 4043/10000 [28:01<17:49,  5.57it/s]
 40%|████      | 4044/10000 [28:01<17:44,  5.60it/s]
 40%|████      | 4045/10000 [28:02<17:43,  5.60it/s]
 40%|████      | 4046/10000 [28:02<18:03,  5.50it/s]
 40%|████      | 4047/10000 [28:02<18:12,  5.45it/s]
 40%|████      | 4048/10000 [28:02<18:07,  5.47it/s]
 40%|████      | 4049/10000 [28:02<17:47,  5.57it/s]
 40%|████      | 4050/10000 [28:03<17:39,  5.62it/s]
                                                    
{'loss': 0.0946, 'grad_norm': 0.9207908511161804, 'learning_rate': 6.934020719040056e-05}

 40%|████      | 4050/10000 [28:03<17:39,  5.62it/s]
 41%|████      | 4051/10000 [28:03<17:49,  5.56it/s]
 41%|████      | 4052/10000 [28:03<17:46,  5.58it/s]
 41%|████      | 4053/10000 [28:03<17:46,  5.58it/s]
 41%|████      | 4054/10000 [28:03<17:44,  5.59it/s]
 41%|████      | 4055/10000 [28:03<17:44,  5.58it/s]
 41%|████      | 4056/10000 [28:04<17:44,  5.59it/s]
 41%|████      | 4057/10000 [28:04<17:37,  5.62it/s]
 41%|████      | 4058/10000 [28:04<17:34,  5.63it/s]
 41%|████      | 4059/10000 [28:04<17:39,  5.61it/s]
 41%|████      | 4060/10000 [28:04<17:36,  5.62it/s]
                                                    
{'loss': 0.0991, 'grad_norm': 0.864277720451355, 'learning_rate': 6.918762503212848e-05}

 41%|████      | 4060/10000 [28:04<17:36,  5.62it/s]
 41%|████      | 4061/10000 [28:05<17:58,  5.50it/s]
 41%|████      | 4062/10000 [28:05<17:58,  5.51it/s]
 41%|████      | 4063/10000 [28:05<18:11,  5.44it/s]
 41%|████      | 4064/10000 [28:05<18:02,  5.49it/s]
 41%|████      | 4065/10000 [28:05<18:20,  5.39it/s]
 41%|████      | 4066/10000 [28:05<18:24,  5.37it/s]
 41%|████      | 4067/10000 [28:06<18:12,  5.43it/s]
 41%|████      | 4068/10000 [28:06<18:34,  5.32it/s]
 41%|████      | 4069/10000 [28:06<19:15,  5.13it/s]
 41%|████      | 4070/10000 [28:06<19:45,  5.00it/s]
                                                    
{'loss': 0.1062, 'grad_norm': 0.8690436482429504, 'learning_rate': 6.903483304106319e-05}

 41%|████      | 4070/10000 [28:06<19:45,  5.00it/s]
 41%|████      | 4071/10000 [28:07<20:26,  4.83it/s]
 41%|████      | 4072/10000 [28:07<19:27,  5.08it/s]
 41%|████      | 4073/10000 [28:07<20:38,  4.79it/s]
 41%|████      | 4074/10000 [28:07<21:30,  4.59it/s]
 41%|████      | 4075/10000 [28:07<22:33,  4.38it/s]
 41%|████      | 4076/10000 [28:08<25:07,  3.93it/s]
 41%|████      | 4077/10000 [28:08<24:47,  3.98it/s]
 41%|████      | 4078/10000 [28:08<22:56,  4.30it/s]
 41%|████      | 4079/10000 [28:08<21:45,  4.54it/s]
 41%|████      | 4080/10000 [28:09<21:06,  4.67it/s]
                                                    
{'loss': 0.1077, 'grad_norm': 1.0050075054168701, 'learning_rate': 6.888183288811341e-05}

 41%|████      | 4080/10000 [28:09<21:06,  4.67it/s]
 41%|████      | 4081/10000 [28:09<21:54,  4.50it/s]
 41%|████      | 4082/10000 [28:09<21:51,  4.51it/s]
 41%|████      | 4083/10000 [28:09<22:23,  4.40it/s]
 41%|████      | 4084/10000 [28:09<22:07,  4.46it/s]
 41%|████      | 4085/10000 [28:10<21:51,  4.51it/s]
 41%|████      | 4086/10000 [28:10<21:33,  4.57it/s]
 41%|████      | 4087/10000 [28:10<20:31,  4.80it/s]
 41%|████      | 4088/10000 [28:10<20:09,  4.89it/s]
 41%|████      | 4089/10000 [28:10<20:03,  4.91it/s]
 41%|████      | 4090/10000 [28:11<20:46,  4.74it/s]
                                                    
{'loss': 0.1294, 'grad_norm': 0.7709590792655945, 'learning_rate': 6.87286262464643e-05}

 41%|████      | 4090/10000 [28:11<20:46,  4.74it/s]
 41%|████      | 4091/10000 [28:11<20:57,  4.70it/s]
 41%|████      | 4092/10000 [28:11<21:48,  4.52it/s]
 41%|████      | 4093/10000 [28:11<21:41,  4.54it/s]
 41%|████      | 4094/10000 [28:12<21:42,  4.54it/s]
 41%|████      | 4095/10000 [28:12<21:43,  4.53it/s]
 41%|████      | 4096/10000 [28:12<21:38,  4.55it/s]
 41%|████      | 4097/10000 [28:12<22:04,  4.46it/s]
 41%|████      | 4098/10000 [28:13<22:15,  4.42it/s]
 41%|████      | 4099/10000 [28:13<22:16,  4.42it/s]
 41%|████      | 4100/10000 [28:13<22:44,  4.32it/s]
                                                    
{'loss': 0.0873, 'grad_norm': 0.8450242877006531, 'learning_rate': 6.857521479155915e-05}

 41%|████      | 4100/10000 [28:13<22:44,  4.32it/s]
 41%|████      | 4101/10000 [28:13<23:48,  4.13it/s]
 41%|████      | 4102/10000 [28:14<24:59,  3.93it/s]
 41%|████      | 4103/10000 [28:14<25:20,  3.88it/s]
 41%|████      | 4104/10000 [28:14<26:02,  3.77it/s]
 41%|████      | 4105/10000 [28:14<25:07,  3.91it/s]
 41%|████      | 4106/10000 [28:15<23:45,  4.13it/s]
 41%|████      | 4107/10000 [28:15<22:52,  4.29it/s]
 41%|████      | 4108/10000 [28:15<21:49,  4.50it/s]
 41%|████      | 4109/10000 [28:15<21:41,  4.53it/s]
 41%|████      | 4110/10000 [28:15<22:08,  4.43it/s]
                                                    
{'loss': 0.1166, 'grad_norm': 0.8444051742553711, 'learning_rate': 6.842160020108104e-05}

 41%|████      | 4110/10000 [28:15<22:08,  4.43it/s]
 41%|████      | 4111/10000 [28:16<22:52,  4.29it/s]
 41%|████      | 4112/10000 [28:16<23:51,  4.11it/s]
 41%|████      | 4113/10000 [28:16<22:47,  4.31it/s]
 41%|████      | 4114/10000 [28:16<21:48,  4.50it/s]
 41%|████      | 4115/10000 [28:16<20:44,  4.73it/s]
 41%|████      | 4116/10000 [28:17<20:15,  4.84it/s]
 41%|████      | 4117/10000 [28:17<20:29,  4.79it/s]
 41%|████      | 4118/10000 [28:17<21:03,  4.66it/s]
 41%|████      | 4119/10000 [28:17<21:49,  4.49it/s]
 41%|████      | 4120/10000 [28:18<22:06,  4.43it/s]
                                                    
{'loss': 0.1066, 'grad_norm': 0.944047212600708, 'learning_rate': 6.826778415493455e-05}

 41%|████      | 4120/10000 [28:18<22:06,  4.43it/s]
 41%|████      | 4121/10000 [28:18<21:41,  4.52it/s]
 41%|████      | 4122/10000 [28:18<21:20,  4.59it/s]
 41%|████      | 4123/10000 [28:18<20:58,  4.67it/s]
 41%|████      | 4124/10000 [28:18<21:19,  4.59it/s]
 41%|████▏     | 4125/10000 [28:19<21:16,  4.60it/s]
 41%|████▏     | 4126/10000 [28:19<21:41,  4.51it/s]
 41%|████▏     | 4127/10000 [28:19<22:17,  4.39it/s]
 41%|████▏     | 4128/10000 [28:19<22:30,  4.35it/s]
 41%|████▏     | 4129/10000 [28:20<21:57,  4.46it/s]
 41%|████▏     | 4130/10000 [28:20<21:12,  4.61it/s]
                                                    
{'loss': 0.0953, 'grad_norm': 0.8900109529495239, 'learning_rate': 6.811376833522729e-05}

 41%|████▏     | 4130/10000 [28:20<21:12,  4.61it/s]
 41%|████▏     | 4131/10000 [28:20<21:03,  4.65it/s]
 41%|████▏     | 4132/10000 [28:20<20:34,  4.75it/s]
 41%|████▏     | 4133/10000 [28:20<20:43,  4.72it/s]
 41%|████▏     | 4134/10000 [28:21<21:21,  4.58it/s]
 41%|████▏     | 4135/10000 [28:21<21:26,  4.56it/s]
 41%|████▏     | 4136/10000 [28:21<22:02,  4.43it/s]
 41%|████▏     | 4137/10000 [28:21<22:30,  4.34it/s]
 41%|████▏     | 4138/10000 [28:22<21:50,  4.47it/s]
 41%|████▏     | 4139/10000 [28:22<20:57,  4.66it/s]
 41%|████▏     | 4140/10000 [28:22<20:36,  4.74it/s]
                                                    
{'loss': 0.0959, 'grad_norm': 1.1624770164489746, 'learning_rate': 6.795955442625159e-05}

 41%|████▏     | 4140/10000 [28:22<20:36,  4.74it/s]
 41%|████▏     | 4141/10000 [28:22<20:53,  4.67it/s]
 41%|████▏     | 4142/10000 [28:22<21:11,  4.61it/s]
 41%|████▏     | 4143/10000 [28:23<21:32,  4.53it/s]
 41%|████▏     | 4144/10000 [28:23<21:31,  4.53it/s]
 41%|████▏     | 4145/10000 [28:23<20:51,  4.68it/s]
 41%|████▏     | 4146/10000 [28:23<20:18,  4.81it/s]
 41%|████▏     | 4147/10000 [28:23<19:55,  4.89it/s]
 41%|████▏     | 4148/10000 [28:24<19:12,  5.08it/s]
 41%|████▏     | 4149/10000 [28:24<18:50,  5.17it/s]
 42%|████▏     | 4150/10000 [28:24<19:26,  5.01it/s]
                                                    
{'loss': 0.1145, 'grad_norm': 0.7927098274230957, 'learning_rate': 6.780514411446608e-05}

 42%|████▏     | 4150/10000 [28:24<19:26,  5.01it/s]
 42%|████▏     | 4151/10000 [28:24<20:40,  4.71it/s]
 42%|████▏     | 4152/10000 [28:24<21:35,  4.51it/s]
 42%|████▏     | 4153/10000 [28:25<22:32,  4.32it/s]
 42%|████▏     | 4154/10000 [28:25<23:01,  4.23it/s]
 42%|████▏     | 4155/10000 [28:25<22:11,  4.39it/s]
 42%|████▏     | 4156/10000 [28:25<21:02,  4.63it/s]
 42%|████▏     | 4157/10000 [28:26<19:59,  4.87it/s]
 42%|████▏     | 4158/10000 [28:26<19:29,  5.00it/s]
 42%|████▏     | 4159/10000 [28:26<19:41,  4.95it/s]
 42%|████▏     | 4160/10000 [28:26<20:30,  4.75it/s]
                                                    
{'loss': 0.1156, 'grad_norm': 0.9106976985931396, 'learning_rate': 6.765053908847716e-05}

 42%|████▏     | 4160/10000 [28:26<20:30,  4.75it/s]
 42%|████▏     | 4161/10000 [28:26<20:42,  4.70it/s]
 42%|████▏     | 4162/10000 [28:27<20:49,  4.67it/s]
 42%|████▏     | 4163/10000 [28:27<21:01,  4.63it/s]
 42%|████▏     | 4164/10000 [28:27<21:02,  4.62it/s]
 42%|████▏     | 4165/10000 [28:27<20:22,  4.77it/s]
 42%|████▏     | 4166/10000 [28:27<19:42,  4.94it/s]
 42%|████▏     | 4167/10000 [28:28<19:10,  5.07it/s]
 42%|████▏     | 4168/10000 [28:28<18:45,  5.18it/s]
 42%|████▏     | 4169/10000 [28:28<19:24,  5.01it/s]
 42%|████▏     | 4170/10000 [28:28<20:17,  4.79it/s]
                                                    
{'loss': 0.1125, 'grad_norm': 0.8723145127296448, 'learning_rate': 6.749574103902064e-05}

 42%|████▏     | 4170/10000 [28:28<20:17,  4.79it/s]
 42%|████▏     | 4171/10000 [28:29<20:59,  4.63it/s]
 42%|████▏     | 4172/10000 [28:29<21:14,  4.57it/s]
 42%|████▏     | 4173/10000 [28:29<21:12,  4.58it/s]
 42%|████▏     | 4174/10000 [28:29<20:55,  4.64it/s]
 42%|████▏     | 4175/10000 [28:29<21:05,  4.60it/s]
 42%|████▏     | 4176/10000 [28:30<20:37,  4.71it/s]
 42%|████▏     | 4177/10000 [28:30<20:47,  4.67it/s]
 42%|████▏     | 4178/10000 [28:30<21:09,  4.59it/s]
 42%|████▏     | 4179/10000 [28:30<21:11,  4.58it/s]
 42%|████▏     | 4180/10000 [28:30<21:53,  4.43it/s]
                                                    
{'loss': 0.1093, 'grad_norm': 0.7838819622993469, 'learning_rate': 6.734075165894317e-05}

 42%|████▏     | 4180/10000 [28:31<21:53,  4.43it/s]
 42%|████▏     | 4181/10000 [28:31<22:43,  4.27it/s]
 42%|████▏     | 4182/10000 [28:31<22:15,  4.36it/s]
 42%|████▏     | 4183/10000 [28:31<21:08,  4.59it/s]
 42%|████▏     | 4184/10000 [28:31<20:03,  4.83it/s]
 42%|████▏     | 4185/10000 [28:32<19:35,  4.95it/s]
 42%|████▏     | 4186/10000 [28:32<19:36,  4.94it/s]
 42%|████▏     | 4187/10000 [28:32<19:59,  4.85it/s]
 42%|████▏     | 4188/10000 [28:32<20:40,  4.69it/s]
 42%|████▏     | 4189/10000 [28:32<21:30,  4.50it/s]
 42%|████▏     | 4190/10000 [28:33<21:11,  4.57it/s]
                                                    
{'loss': 0.1084, 'grad_norm': 0.6333476901054382, 'learning_rate': 6.71855726431838e-05}

 42%|████▏     | 4190/10000 [28:33<21:11,  4.57it/s]
 42%|████▏     | 4191/10000 [28:33<21:09,  4.58it/s]
 42%|████▏     | 4192/10000 [28:33<20:12,  4.79it/s]
 42%|████▏     | 4193/10000 [28:33<20:05,  4.82it/s]
 42%|████▏     | 4194/10000 [28:33<20:20,  4.76it/s]
 42%|████▏     | 4195/10000 [28:34<20:56,  4.62it/s]
 42%|████▏     | 4196/10000 [28:34<21:11,  4.56it/s]
 42%|████▏     | 4197/10000 [28:34<21:19,  4.54it/s]
 42%|████▏     | 4198/10000 [28:34<20:54,  4.62it/s]
 42%|████▏     | 4199/10000 [28:35<20:08,  4.80it/s]
 42%|████▏     | 4200/10000 [28:35<19:21,  4.99it/s]
                                                    
{'loss': 0.1083, 'grad_norm': 0.7201565504074097, 'learning_rate': 6.703020568875538e-05}

 42%|████▏     | 4200/10000 [28:35<19:21,  4.99it/s]
 42%|████▏     | 4201/10000 [28:35<19:17,  5.01it/s]
 42%|████▏     | 4202/10000 [28:35<19:45,  4.89it/s]
 42%|████▏     | 4203/10000 [28:35<20:30,  4.71it/s]
 42%|████▏     | 4204/10000 [28:36<21:07,  4.57it/s]
 42%|████▏     | 4205/10000 [28:36<21:26,  4.50it/s]
 42%|████▏     | 4206/10000 [28:36<20:53,  4.62it/s]
 42%|████▏     | 4207/10000 [28:36<20:10,  4.79it/s]
 42%|████▏     | 4208/10000 [28:36<19:37,  4.92it/s]
 42%|████▏     | 4209/10000 [28:37<19:11,  5.03it/s]
 42%|████▏     | 4210/10000 [28:37<20:03,  4.81it/s]
                                                    
{'loss': 0.0952, 'grad_norm': 0.8311694860458374, 'learning_rate': 6.687465249472603e-05}

 42%|████▏     | 4210/10000 [28:37<20:03,  4.81it/s]
 42%|████▏     | 4211/10000 [28:37<21:11,  4.55it/s]
 42%|████▏     | 4212/10000 [28:37<21:41,  4.45it/s]
 42%|████▏     | 4213/10000 [28:38<21:16,  4.53it/s]
 42%|████▏     | 4214/10000 [28:38<20:39,  4.67it/s]
 42%|████▏     | 4215/10000 [28:38<19:41,  4.90it/s]
 42%|████▏     | 4216/10000 [28:38<19:21,  4.98it/s]
 42%|████▏     | 4217/10000 [28:38<19:37,  4.91it/s]
 42%|████▏     | 4218/10000 [28:39<20:20,  4.74it/s]
 42%|████▏     | 4219/10000 [28:39<20:23,  4.72it/s]
 42%|████▏     | 4220/10000 [28:39<20:47,  4.63it/s]
                                                    
{'loss': 0.1052, 'grad_norm': 0.8421615362167358, 'learning_rate': 6.671891476220055e-05}

 42%|████▏     | 4220/10000 [28:39<20:47,  4.63it/s]
 42%|████▏     | 4221/10000 [28:39<20:45,  4.64it/s]
 42%|████▏     | 4222/10000 [28:39<19:34,  4.92it/s]
 42%|████▏     | 4223/10000 [28:40<18:43,  5.14it/s]
 42%|████▏     | 4224/10000 [28:40<18:20,  5.25it/s]
 42%|████▏     | 4225/10000 [28:40<18:49,  5.11it/s]
 42%|████▏     | 4226/10000 [28:40<19:39,  4.90it/s]
 42%|████▏     | 4227/10000 [28:40<20:03,  4.80it/s]
 42%|████▏     | 4228/10000 [28:41<20:16,  4.75it/s]
 42%|████▏     | 4229/10000 [28:41<19:42,  4.88it/s]
 42%|████▏     | 4230/10000 [28:41<19:04,  5.04it/s]
                                                    
{'loss': 0.0841, 'grad_norm': 0.9531618356704712, 'learning_rate': 6.656299419430183e-05}

 42%|████▏     | 4230/10000 [28:41<19:04,  5.04it/s]
 42%|████▏     | 4231/10000 [28:41<18:39,  5.15it/s]
 42%|████▏     | 4232/10000 [28:41<18:09,  5.29it/s]
 42%|████▏     | 4233/10000 [28:41<17:58,  5.35it/s]
 42%|████▏     | 4234/10000 [28:42<17:50,  5.39it/s]
 42%|████▏     | 4235/10000 [28:42<18:21,  5.23it/s]
 42%|████▏     | 4236/10000 [28:42<18:34,  5.17it/s]
 42%|████▏     | 4237/10000 [28:42<18:44,  5.13it/s]
 42%|████▏     | 4238/10000 [28:42<18:33,  5.17it/s]
 42%|████▏     | 4239/10000 [28:43<18:06,  5.30it/s]
 42%|████▏     | 4240/10000 [28:43<17:37,  5.44it/s]
                                                    
{'loss': 0.0944, 'grad_norm': 0.8325589895248413, 'learning_rate': 6.640689249615223e-05}

 42%|████▏     | 4240/10000 [28:43<17:37,  5.44it/s]
 42%|████▏     | 4241/10000 [28:43<17:33,  5.46it/s]
 42%|████▏     | 4242/10000 [28:43<17:31,  5.48it/s]
 42%|████▏     | 4243/10000 [28:43<17:50,  5.38it/s]
 42%|████▏     | 4244/10000 [28:44<18:18,  5.24it/s]
 42%|████▏     | 4245/10000 [28:44<18:46,  5.11it/s]
 42%|████▏     | 4246/10000 [28:44<18:41,  5.13it/s]
 42%|████▏     | 4247/10000 [28:44<18:34,  5.16it/s]
 42%|████▏     | 4248/10000 [28:44<18:34,  5.16it/s]
 42%|████▏     | 4249/10000 [28:45<18:31,  5.17it/s]
 42%|████▎     | 4250/10000 [28:45<18:40,  5.13it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.16 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 16.39 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.44 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 16.66 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 16.83 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 16.93 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 4250: eval/loss = 0.1035 (34s)

                                                    
{'loss': 0.0991, 'grad_norm': 0.8695216774940491, 'learning_rate': 6.625061137485491e-05}

 42%|████▎     | 4250/10000 [29:19<18:40,  5.13it/s]
 43%|████▎     | 4251/10000 [29:19<16:48:32, 10.53s/it]
 43%|████▎     | 4252/10000 [29:20<11:51:07,  7.42s/it]
 43%|████▎     | 4253/10000 [29:20<8:22:46,  5.25s/it] 
 43%|████▎     | 4254/10000 [29:20<5:56:57,  3.73s/it]
 43%|████▎     | 4255/10000 [29:20<4:15:09,  2.66s/it]
 43%|████▎     | 4256/10000 [29:20<3:03:40,  1.92s/it]
 43%|████▎     | 4257/10000 [29:20<2:13:56,  1.40s/it]
 43%|████▎     | 4258/10000 [29:21<1:39:17,  1.04s/it]
 43%|████▎     | 4259/10000 [29:21<1:14:41,  1.28it/s]
 43%|████▎     | 4260/10000 [29:21<57:27,  1.67it/s]  
                                                    
{'loss': 0.1121, 'grad_norm': 0.9460748434066772, 'learning_rate': 6.609415253947517e-05}

 43%|████▎     | 4260/10000 [29:21<57:27,  1.67it/s]
 43%|████▎     | 4261/10000 [29:21<45:46,  2.09it/s]
 43%|████▎     | 4262/10000 [29:21<37:13,  2.57it/s]
 43%|████▎     | 4263/10000 [29:22<31:15,  3.06it/s]
 43%|████▎     | 4264/10000 [29:22<27:05,  3.53it/s]
 43%|████▎     | 4265/10000 [29:22<23:58,  3.99it/s]
 43%|████▎     | 4266/10000 [29:22<22:04,  4.33it/s]
 43%|████▎     | 4267/10000 [29:22<21:22,  4.47it/s]
 43%|████▎     | 4268/10000 [29:23<20:44,  4.61it/s]
 43%|████▎     | 4269/10000 [29:23<20:04,  4.76it/s]
 43%|████▎     | 4270/10000 [29:23<19:56,  4.79it/s]
                                                    
{'loss': 0.1152, 'grad_norm': 0.7062116265296936, 'learning_rate': 6.593751770102178e-05}

 43%|████▎     | 4270/10000 [29:23<19:56,  4.79it/s]
 43%|████▎     | 4271/10000 [29:23<19:14,  4.96it/s]
 43%|████▎     | 4272/10000 [29:23<18:30,  5.16it/s]
 43%|████▎     | 4273/10000 [29:23<18:21,  5.20it/s]
 43%|████▎     | 4274/10000 [29:24<18:07,  5.26it/s]
 43%|████▎     | 4275/10000 [29:24<17:56,  5.32it/s]
 43%|████▎     | 4276/10000 [29:24<17:50,  5.35it/s]
 43%|████▎     | 4277/10000 [29:24<17:32,  5.44it/s]
 43%|████▎     | 4278/10000 [29:24<17:17,  5.51it/s]
 43%|████▎     | 4279/10000 [29:25<17:15,  5.52it/s]
 43%|████▎     | 4280/10000 [29:25<17:01,  5.60it/s]
                                                    
{'loss': 0.1114, 'grad_norm': 1.137629508972168, 'learning_rate': 6.578070857242823e-05}

 43%|████▎     | 4280/10000 [29:25<17:01,  5.60it/s]
 43%|████▎     | 4281/10000 [29:25<17:13,  5.53it/s]
 43%|████▎     | 4282/10000 [29:25<17:05,  5.58it/s]
 43%|████▎     | 4283/10000 [29:25<17:02,  5.59it/s]
 43%|████▎     | 4284/10000 [29:25<17:02,  5.59it/s]
 43%|████▎     | 4285/10000 [29:26<17:19,  5.50it/s]
 43%|████▎     | 4286/10000 [29:26<17:13,  5.53it/s]
 43%|████▎     | 4287/10000 [29:26<17:03,  5.58it/s]
 43%|████▎     | 4288/10000 [29:26<17:28,  5.45it/s]
 43%|████▎     | 4289/10000 [29:26<17:13,  5.52it/s]
 43%|████▎     | 4290/10000 [29:27<16:58,  5.61it/s]
                                                    
{'loss': 0.103, 'grad_norm': 0.9860761761665344, 'learning_rate': 6.562372686853402e-05}

 43%|████▎     | 4290/10000 [29:27<16:58,  5.61it/s]
 43%|████▎     | 4291/10000 [29:27<17:12,  5.53it/s]
 43%|████▎     | 4292/10000 [29:27<17:01,  5.59it/s]
 43%|████▎     | 4293/10000 [29:27<16:54,  5.63it/s]
 43%|████▎     | 4294/10000 [29:27<16:54,  5.62it/s]
 43%|████▎     | 4295/10000 [29:27<16:52,  5.63it/s]
 43%|████▎     | 4296/10000 [29:28<16:47,  5.66it/s]
 43%|████▎     | 4297/10000 [29:28<16:50,  5.65it/s]
 43%|████▎     | 4298/10000 [29:28<17:02,  5.57it/s]
 43%|████▎     | 4299/10000 [29:28<16:53,  5.63it/s]
 43%|████▎     | 4300/10000 [29:28<17:27,  5.44it/s]
                                                    
{'loss': 0.0888, 'grad_norm': 0.8624077439308167, 'learning_rate': 6.546657430606593e-05}

 43%|████▎     | 4300/10000 [29:28<17:27,  5.44it/s]
 43%|████▎     | 4301/10000 [29:29<19:24,  4.89it/s]
 43%|████▎     | 4302/10000 [29:29<20:29,  4.63it/s]
 43%|████▎     | 4303/10000 [29:29<20:21,  4.67it/s]
 43%|████▎     | 4304/10000 [29:29<20:40,  4.59it/s]
 43%|████▎     | 4305/10000 [29:29<20:46,  4.57it/s]
 43%|████▎     | 4306/10000 [29:30<20:39,  4.60it/s]
 43%|████▎     | 4307/10000 [29:30<20:38,  4.60it/s]
 43%|████▎     | 4308/10000 [29:30<20:48,  4.56it/s]
 43%|████▎     | 4309/10000 [29:30<20:19,  4.67it/s]
 43%|████▎     | 4310/10000 [29:31<20:12,  4.69it/s]
                                                    
{'loss': 0.135, 'grad_norm': 1.012800693511963, 'learning_rate': 6.530925260361918e-05}

 43%|████▎     | 4310/10000 [29:31<20:12,  4.69it/s]
 43%|████▎     | 4311/10000 [29:31<19:44,  4.80it/s]
 43%|████▎     | 4312/10000 [29:31<19:18,  4.91it/s]
 43%|████▎     | 4313/10000 [29:31<18:48,  5.04it/s]
 43%|████▎     | 4314/10000 [29:31<18:51,  5.03it/s]
 43%|████▎     | 4315/10000 [29:32<18:36,  5.09it/s]
 43%|████▎     | 4316/10000 [29:32<18:12,  5.20it/s]
 43%|████▎     | 4317/10000 [29:32<17:57,  5.28it/s]
 43%|████▎     | 4318/10000 [29:32<17:40,  5.36it/s]
 43%|████▎     | 4319/10000 [29:32<17:46,  5.33it/s]
 43%|████▎     | 4320/10000 [29:32<17:55,  5.28it/s]
                                                    
{'loss': 0.0833, 'grad_norm': 0.9095959663391113, 'learning_rate': 6.515176348163871e-05}

 43%|████▎     | 4320/10000 [29:32<17:55,  5.28it/s]
 43%|████▎     | 4321/10000 [29:33<18:40,  5.07it/s]
 43%|████▎     | 4322/10000 [29:33<18:48,  5.03it/s]
 43%|████▎     | 4323/10000 [29:33<18:54,  5.01it/s]
 43%|████▎     | 4324/10000 [29:33<18:30,  5.11it/s]
 43%|████▎     | 4325/10000 [29:33<18:20,  5.16it/s]
 43%|████▎     | 4326/10000 [29:34<18:05,  5.23it/s]
 43%|████▎     | 4327/10000 [29:34<17:41,  5.35it/s]
 43%|████▎     | 4328/10000 [29:34<17:46,  5.32it/s]
 43%|████▎     | 4329/10000 [29:34<17:51,  5.29it/s]
 43%|████▎     | 4330/10000 [29:34<17:57,  5.26it/s]
                                                    
{'loss': 0.0901, 'grad_norm': 0.8667500019073486, 'learning_rate': 6.499410866240032e-05}

 43%|████▎     | 4330/10000 [29:34<17:57,  5.26it/s]
 43%|████▎     | 4331/10000 [29:35<18:18,  5.16it/s]
 43%|████▎     | 4332/10000 [29:35<18:23,  5.14it/s]
 43%|████▎     | 4333/10000 [29:35<18:19,  5.15it/s]
 43%|████▎     | 4334/10000 [29:35<18:16,  5.17it/s]
 43%|████▎     | 4335/10000 [29:35<18:23,  5.13it/s]
 43%|████▎     | 4336/10000 [29:36<18:14,  5.18it/s]
 43%|████▎     | 4337/10000 [29:36<18:08,  5.20it/s]
 43%|████▎     | 4338/10000 [29:36<18:29,  5.10it/s]
 43%|████▎     | 4339/10000 [29:36<18:14,  5.17it/s]
 43%|████▎     | 4340/10000 [29:36<17:55,  5.26it/s]
                                                    
{'loss': 0.1272, 'grad_norm': 0.9169116616249084, 'learning_rate': 6.48362898699919e-05}

 43%|████▎     | 4340/10000 [29:36<17:55,  5.26it/s]
 43%|████▎     | 4341/10000 [29:37<18:00,  5.24it/s]
 43%|████▎     | 4342/10000 [29:37<17:34,  5.37it/s]
 43%|████▎     | 4343/10000 [29:37<17:26,  5.41it/s]
 43%|████▎     | 4344/10000 [29:37<17:47,  5.30it/s]
 43%|████▎     | 4345/10000 [29:37<17:52,  5.27it/s]
 43%|████▎     | 4346/10000 [29:37<18:11,  5.18it/s]
 43%|████▎     | 4347/10000 [29:38<18:24,  5.12it/s]
 43%|████▎     | 4348/10000 [29:38<18:10,  5.18it/s]
 43%|████▎     | 4349/10000 [29:38<18:11,  5.18it/s]
 44%|████▎     | 4350/10000 [29:38<18:08,  5.19it/s]
                                                    
{'loss': 0.0892, 'grad_norm': 1.095344066619873, 'learning_rate': 6.467830883029443e-05}

 44%|████▎     | 4350/10000 [29:38<18:08,  5.19it/s]
 44%|████▎     | 4351/10000 [29:38<17:58,  5.24it/s]
 44%|████▎     | 4352/10000 [29:39<17:56,  5.25it/s]
 44%|████▎     | 4353/10000 [29:39<18:26,  5.10it/s]
 44%|████▎     | 4354/10000 [29:39<18:23,  5.12it/s]
 44%|████▎     | 4355/10000 [29:39<18:11,  5.17it/s]
 44%|████▎     | 4356/10000 [29:39<18:03,  5.21it/s]
 44%|████▎     | 4357/10000 [29:40<18:00,  5.22it/s]
 44%|████▎     | 4358/10000 [29:40<17:57,  5.24it/s]
 44%|████▎     | 4359/10000 [29:40<17:55,  5.24it/s]
 44%|████▎     | 4360/10000 [29:40<17:44,  5.30it/s]
                                                    
{'loss': 0.1063, 'grad_norm': 0.8654522895812988, 'learning_rate': 6.452016727096326e-05}

 44%|████▎     | 4360/10000 [29:40<17:44,  5.30it/s]
 44%|████▎     | 4361/10000 [29:40<17:45,  5.29it/s]
 44%|████▎     | 4362/10000 [29:41<17:54,  5.25it/s]
 44%|████▎     | 4363/10000 [29:41<18:09,  5.17it/s]
 44%|████▎     | 4364/10000 [29:41<18:17,  5.13it/s]
 44%|████▎     | 4365/10000 [29:41<18:27,  5.09it/s]
 44%|████▎     | 4366/10000 [29:41<18:31,  5.07it/s]
 44%|████▎     | 4367/10000 [29:42<18:24,  5.10it/s]
 44%|████▎     | 4368/10000 [29:42<18:00,  5.21it/s]
 44%|████▎     | 4369/10000 [29:42<17:45,  5.29it/s]
 44%|████▎     | 4370/10000 [29:42<17:34,  5.34it/s]
                                                    
{'loss': 0.0923, 'grad_norm': 0.9861807227134705, 'learning_rate': 6.436186692140916e-05}

 44%|████▎     | 4370/10000 [29:42<17:34,  5.34it/s]
 44%|████▎     | 4371/10000 [29:42<17:47,  5.27it/s]
 44%|████▎     | 4372/10000 [29:42<18:01,  5.20it/s]
 44%|████▎     | 4373/10000 [29:43<18:11,  5.16it/s]
 44%|████▎     | 4374/10000 [29:43<18:16,  5.13it/s]
 44%|████▍     | 4375/10000 [29:43<18:17,  5.13it/s]
 44%|████▍     | 4376/10000 [29:43<17:59,  5.21it/s]
 44%|████▍     | 4377/10000 [29:43<17:41,  5.30it/s]
 44%|████▍     | 4378/10000 [29:44<17:23,  5.39it/s]
 44%|████▍     | 4379/10000 [29:44<20:32,  4.56it/s]
 44%|████▍     | 4380/10000 [29:44<19:38,  4.77it/s]
                                                    
{'loss': 0.0915, 'grad_norm': 0.8531836271286011, 'learning_rate': 6.420340951277938e-05}

 44%|████▍     | 4380/10000 [29:44<19:38,  4.77it/s]
 44%|████▍     | 4381/10000 [29:44<19:09,  4.89it/s]
 44%|████▍     | 4382/10000 [29:44<18:51,  4.97it/s]
 44%|████▍     | 4383/10000 [29:45<18:09,  5.16it/s]
 44%|████▍     | 4384/10000 [29:45<17:59,  5.20it/s]
 44%|████▍     | 4385/10000 [29:45<17:49,  5.25it/s]
 44%|████▍     | 4386/10000 [29:45<17:54,  5.22it/s]
 44%|████▍     | 4387/10000 [29:45<17:50,  5.24it/s]
 44%|████▍     | 4388/10000 [29:46<17:42,  5.28it/s]
 44%|████▍     | 4389/10000 [29:46<17:27,  5.36it/s]
 44%|████▍     | 4390/10000 [29:46<17:19,  5.40it/s]
                                                    
{'loss': 0.1058, 'grad_norm': 0.7771260142326355, 'learning_rate': 6.404479677793874e-05}

 44%|████▍     | 4390/10000 [29:46<17:19,  5.40it/s]
 44%|████▍     | 4391/10000 [29:46<17:28,  5.35it/s]
 44%|████▍     | 4392/10000 [29:46<17:12,  5.43it/s]
 44%|████▍     | 4393/10000 [29:46<17:07,  5.46it/s]
 44%|████▍     | 4394/10000 [29:47<16:57,  5.51it/s]
 44%|████▍     | 4395/10000 [29:47<17:03,  5.48it/s]
 44%|████▍     | 4396/10000 [29:47<16:54,  5.52it/s]
 44%|████▍     | 4397/10000 [29:47<16:56,  5.51it/s]
 44%|████▍     | 4398/10000 [29:47<16:49,  5.55it/s]
 44%|████▍     | 4399/10000 [29:48<16:37,  5.61it/s]
 44%|████▍     | 4400/10000 [29:48<16:47,  5.56it/s]
                                                    
{'loss': 0.1123, 'grad_norm': 0.8296533823013306, 'learning_rate': 6.388603045145075e-05}

 44%|████▍     | 4400/10000 [29:48<16:47,  5.56it/s]
 44%|████▍     | 4401/10000 [29:48<17:26,  5.35it/s]
 44%|████▍     | 4402/10000 [29:48<17:10,  5.43it/s]
 44%|████▍     | 4403/10000 [29:48<17:16,  5.40it/s]
 44%|████▍     | 4404/10000 [29:49<17:16,  5.40it/s]
 44%|████▍     | 4405/10000 [29:49<17:17,  5.39it/s]
 44%|████▍     | 4406/10000 [29:49<17:05,  5.46it/s]
 44%|████▍     | 4407/10000 [29:49<16:54,  5.51it/s]
 44%|████▍     | 4408/10000 [29:49<16:39,  5.59it/s]
 44%|████▍     | 4409/10000 [29:49<16:42,  5.57it/s]
 44%|████▍     | 4410/10000 [29:50<16:44,  5.56it/s]
                                                    
{'loss': 0.0913, 'grad_norm': 0.961249053478241, 'learning_rate': 6.372711226955843e-05}

 44%|████▍     | 4410/10000 [29:50<16:44,  5.56it/s]
 44%|████▍     | 4411/10000 [29:50<17:15,  5.40it/s]
 44%|████▍     | 4412/10000 [29:50<17:24,  5.35it/s]
 44%|████▍     | 4413/10000 [29:50<17:12,  5.41it/s]
 44%|████▍     | 4414/10000 [29:50<16:57,  5.49it/s]
 44%|████▍     | 4415/10000 [29:51<16:51,  5.52it/s]
 44%|████▍     | 4416/10000 [29:51<16:39,  5.58it/s]
 44%|████▍     | 4417/10000 [29:51<16:44,  5.56it/s]
 44%|████▍     | 4418/10000 [29:51<16:46,  5.54it/s]
 44%|████▍     | 4419/10000 [29:51<16:42,  5.57it/s]
 44%|████▍     | 4420/10000 [29:51<16:43,  5.56it/s]
                                                    
{'loss': 0.1045, 'grad_norm': 0.8659205436706543, 'learning_rate': 6.356804397016564e-05}

 44%|████▍     | 4420/10000 [29:51<16:43,  5.56it/s]
 44%|████▍     | 4421/10000 [29:52<17:07,  5.43it/s]
 44%|████▍     | 4422/10000 [29:52<17:10,  5.41it/s]
 44%|████▍     | 4423/10000 [29:52<17:05,  5.44it/s]
 44%|████▍     | 4424/10000 [29:52<17:06,  5.43it/s]
 44%|████▍     | 4425/10000 [29:52<17:00,  5.46it/s]
 44%|████▍     | 4426/10000 [29:53<16:44,  5.55it/s]
 44%|████▍     | 4427/10000 [29:53<16:37,  5.59it/s]
 44%|████▍     | 4428/10000 [29:53<16:42,  5.56it/s]
 44%|████▍     | 4429/10000 [29:53<16:53,  5.50it/s]
 44%|████▍     | 4430/10000 [29:53<17:02,  5.45it/s]
                                                    
{'loss': 0.0946, 'grad_norm': 0.8407870531082153, 'learning_rate': 6.340882729281779e-05}

 44%|████▍     | 4430/10000 [29:53<17:02,  5.45it/s]
 44%|████▍     | 4431/10000 [29:53<17:10,  5.40it/s]
 44%|████▍     | 4432/10000 [29:54<16:50,  5.51it/s]
 44%|████▍     | 4433/10000 [29:54<16:37,  5.58it/s]
 44%|████▍     | 4434/10000 [29:54<16:34,  5.60it/s]
 44%|████▍     | 4435/10000 [29:54<16:24,  5.65it/s]
 44%|████▍     | 4436/10000 [29:54<16:16,  5.70it/s]
 44%|████▍     | 4437/10000 [29:54<16:21,  5.67it/s]
 44%|████▍     | 4438/10000 [29:55<16:21,  5.67it/s]
 44%|████▍     | 4439/10000 [29:55<16:24,  5.65it/s]
 44%|████▍     | 4440/10000 [29:55<16:59,  5.45it/s]
                                                    
{'loss': 0.1014, 'grad_norm': 0.8015242218971252, 'learning_rate': 6.324946397868294e-05}

 44%|████▍     | 4440/10000 [29:55<16:59,  5.45it/s]
 44%|████▍     | 4441/10000 [29:55<17:16,  5.37it/s]
 44%|████▍     | 4442/10000 [29:55<17:11,  5.39it/s]
 44%|████▍     | 4443/10000 [29:56<17:12,  5.38it/s]
 44%|████▍     | 4444/10000 [29:56<16:52,  5.49it/s]
 44%|████▍     | 4445/10000 [29:56<16:52,  5.49it/s]
 44%|████▍     | 4446/10000 [29:56<16:59,  5.45it/s]
 44%|████▍     | 4447/10000 [29:56<16:45,  5.52it/s]
 44%|████▍     | 4448/10000 [29:57<17:04,  5.42it/s]
 44%|████▍     | 4449/10000 [29:57<17:01,  5.44it/s]
 44%|████▍     | 4450/10000 [29:57<16:50,  5.49it/s]
                                                    
{'loss': 0.0865, 'grad_norm': 0.7973492741584778, 'learning_rate': 6.308995577053276e-05}

 44%|████▍     | 4450/10000 [29:57<16:50,  5.49it/s]
 45%|████▍     | 4451/10000 [29:57<17:05,  5.41it/s]
 45%|████▍     | 4452/10000 [29:57<16:47,  5.51it/s]
 45%|████▍     | 4453/10000 [29:57<16:40,  5.55it/s]
 45%|████▍     | 4454/10000 [29:58<16:29,  5.60it/s]
 45%|████▍     | 4455/10000 [29:58<16:29,  5.60it/s]
 45%|████▍     | 4456/10000 [29:58<16:33,  5.58it/s]
 45%|████▍     | 4457/10000 [29:58<16:53,  5.47it/s]
 45%|████▍     | 4458/10000 [29:58<17:00,  5.43it/s]
 45%|████▍     | 4459/10000 [29:58<16:46,  5.50it/s]
 45%|████▍     | 4460/10000 [29:59<16:38,  5.55it/s]
                                                    
{'loss': 0.0941, 'grad_norm': 0.887520432472229, 'learning_rate': 6.293030441272347e-05}

 45%|████▍     | 4460/10000 [29:59<16:38,  5.55it/s]
 45%|████▍     | 4461/10000 [29:59<16:43,  5.52it/s]
 45%|████▍     | 4462/10000 [29:59<17:44,  5.20it/s]
 45%|████▍     | 4463/10000 [29:59<18:34,  4.97it/s]
 45%|████▍     | 4464/10000 [30:00<19:28,  4.74it/s]
 45%|████▍     | 4465/10000 [30:00<20:22,  4.53it/s]Rank 0, Worker 1: Wait for shard 20 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 45%|████▍     | 4466/10000 [30:00<21:12,  4.35it/s]
 45%|████▍     | 4467/10000 [30:00<20:32,  4.49it/s]
 45%|████▍     | 4468/10000 [30:00<20:02,  4.60it/s]Rank 0, Worker 4: Wait for shard 77 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 45%|████▍     | 4469/10000 [30:01<19:52,  4.64it/s]
 45%|████▍     | 4470/10000 [30:01<19:31,  4.72it/s]
                                                    
{'loss': 0.1007, 'grad_norm': 0.7706654667854309, 'learning_rate': 6.277051165117677e-05}

 45%|████▍     | 4470/10000 [30:01<19:31,  4.72it/s]
 45%|████▍     | 4471/10000 [30:01<19:41,  4.68it/s]
 45%|████▍     | 4472/10000 [30:01<20:38,  4.47it/s]
 45%|████▍     | 4473/10000 [30:02<21:15,  4.33it/s]
 45%|████▍     | 4474/10000 [30:02<20:58,  4.39it/s]
 45%|████▍     | 4475/10000 [30:02<20:19,  4.53it/s]
 45%|████▍     | 4476/10000 [30:02<20:02,  4.59it/s]
 45%|████▍     | 4477/10000 [30:02<19:45,  4.66it/s]
 45%|████▍     | 4478/10000 [30:03<19:24,  4.74it/s]
 45%|████▍     | 4479/10000 [30:03<19:34,  4.70it/s]
 45%|████▍     | 4480/10000 [30:03<19:59,  4.60it/s]
                                                    
{'loss': 0.0963, 'grad_norm': 0.7328978776931763, 'learning_rate': 6.261057923336064e-05}

 45%|████▍     | 4480/10000 [30:03<19:59,  4.60it/s]
 45%|████▍     | 4481/10000 [30:03<20:25,  4.50it/s]
 45%|████▍     | 4482/10000 [30:04<20:41,  4.45it/s]
 45%|████▍     | 4483/10000 [30:04<20:38,  4.45it/s]
 45%|████▍     | 4484/10000 [30:04<20:28,  4.49it/s]
 45%|████▍     | 4485/10000 [30:04<20:31,  4.48it/s]
 45%|████▍     | 4486/10000 [30:04<20:03,  4.58it/s]
 45%|████▍     | 4487/10000 [30:05<19:42,  4.66it/s]
 45%|████▍     | 4488/10000 [30:05<19:58,  4.60it/s]
 45%|████▍     | 4489/10000 [30:05<20:09,  4.56it/s]
 45%|████▍     | 4490/10000 [30:05<20:40,  4.44it/s]
                                                    
{'loss': 0.0955, 'grad_norm': 0.8200104832649231, 'learning_rate': 6.245050890827042e-05}

 45%|████▍     | 4490/10000 [30:05<20:40,  4.44it/s]
 45%|████▍     | 4491/10000 [30:06<21:45,  4.22it/s]
 45%|████▍     | 4492/10000 [30:06<22:26,  4.09it/s]
 45%|████▍     | 4493/10000 [30:06<22:21,  4.11it/s]
 45%|████▍     | 4494/10000 [30:06<21:27,  4.28it/s]
 45%|████▍     | 4495/10000 [30:06<20:35,  4.46it/s]
 45%|████▍     | 4496/10000 [30:07<19:27,  4.71it/s]
 45%|████▍     | 4497/10000 [30:07<18:45,  4.89it/s]
 45%|████▍     | 4498/10000 [30:07<18:42,  4.90it/s]
 45%|████▍     | 4499/10000 [30:07<18:58,  4.83it/s]
 45%|████▌     | 4500/10000 [30:08<20:39,  4.44it/s]Rank 0, Worker 2: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 18.18 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.22 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.34 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.41 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.62 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 20.04 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 4500: eval/loss = 0.1090 (39s)

                                                    
{'loss': 0.0826, 'grad_norm': 0.7310347557067871, 'learning_rate': 6.229030242640952e-05}

 45%|████▌     | 4500/10000 [30:47<20:39,  4.44it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-4500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-4500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-4500/wandb_config.json

 45%|████▌     | 4501/10000 [31:16<31:36:22, 20.69s/it]
 45%|████▌     | 4502/10000 [31:16<22:12:42, 14.54s/it]
 45%|████▌     | 4503/10000 [31:16<15:38:04, 10.24s/it]
 45%|████▌     | 4504/10000 [31:17<11:01:42,  7.22s/it]
 45%|████▌     | 4505/10000 [31:17<7:48:02,  5.11s/it] 
 45%|████▌     | 4506/10000 [31:17<5:32:32,  3.63s/it]
 45%|████▌     | 4507/10000 [31:17<3:58:06,  2.60s/it]
 45%|████▌     | 4508/10000 [31:17<2:52:31,  1.88s/it]
 45%|████▌     | 4509/10000 [31:18<2:06:35,  1.38s/it]
 45%|████▌     | 4510/10000 [31:18<1:34:04,  1.03s/it]
                                                      
{'loss': 0.089, 'grad_norm': 0.7029056549072266, 'learning_rate': 6.212996153977037e-05}

 45%|████▌     | 4510/10000 [31:18<1:34:04,  1.03s/it]
 45%|████▌     | 4511/10000 [31:18<1:11:31,  1.28it/s]
 45%|████▌     | 4512/10000 [31:18<55:07,  1.66it/s]  
 45%|████▌     | 4513/10000 [31:18<43:29,  2.10it/s]
 45%|████▌     | 4514/10000 [31:18<35:23,  2.58it/s]
 45%|████▌     | 4515/10000 [31:19<30:00,  3.05it/s]
 45%|████▌     | 4516/10000 [31:19<26:20,  3.47it/s]
 45%|████▌     | 4517/10000 [31:19<24:15,  3.77it/s]
 45%|████▌     | 4518/10000 [31:19<22:20,  4.09it/s]
 45%|████▌     | 4519/10000 [31:19<21:02,  4.34it/s]
 45%|████▌     | 4520/10000 [31:20<19:54,  4.59it/s]
                                                    
{'loss': 0.0908, 'grad_norm': 0.8050764203071594, 'learning_rate': 6.196948800181523e-05}

 45%|████▌     | 4520/10000 [31:20<19:54,  4.59it/s]
 45%|████▌     | 4521/10000 [31:20<18:58,  4.81it/s]
 45%|████▌     | 4522/10000 [31:20<18:07,  5.04it/s]
 45%|████▌     | 4523/10000 [31:20<17:30,  5.21it/s]
 45%|████▌     | 4524/10000 [31:20<17:07,  5.33it/s]
 45%|████▌     | 4525/10000 [31:21<17:04,  5.35it/s]
 45%|████▌     | 4526/10000 [31:21<16:48,  5.43it/s]
 45%|████▌     | 4527/10000 [31:21<16:25,  5.55it/s]
 45%|████▌     | 4528/10000 [31:21<16:11,  5.63it/s]
 45%|████▌     | 4529/10000 [31:21<15:59,  5.70it/s]
 45%|████▌     | 4530/10000 [31:21<16:00,  5.69it/s]
                                                    
{'loss': 0.0974, 'grad_norm': 0.9031093120574951, 'learning_rate': 6.180888356745695e-05}

 45%|████▌     | 4530/10000 [31:21<16:00,  5.69it/s]
 45%|████▌     | 4531/10000 [31:22<16:11,  5.63it/s]
 45%|████▌     | 4532/10000 [31:22<16:01,  5.68it/s]
 45%|████▌     | 4533/10000 [31:22<15:53,  5.73it/s]
 45%|████▌     | 4534/10000 [31:22<15:58,  5.70it/s]
 45%|████▌     | 4535/10000 [31:22<15:58,  5.70it/s]
 45%|████▌     | 4536/10000 [31:22<15:56,  5.71it/s]
 45%|████▌     | 4537/10000 [31:23<16:04,  5.67it/s]
 45%|████▌     | 4538/10000 [31:23<16:20,  5.57it/s]
 45%|████▌     | 4539/10000 [31:23<16:11,  5.62it/s]
 45%|████▌     | 4540/10000 [31:23<16:10,  5.63it/s]
                                                    
{'loss': 0.1063, 'grad_norm': 0.993007481098175, 'learning_rate': 6.164814999303995e-05}

 45%|████▌     | 4540/10000 [31:23<16:10,  5.63it/s]
 45%|████▌     | 4541/10000 [31:23<16:18,  5.58it/s]
 45%|████▌     | 4542/10000 [31:24<16:08,  5.64it/s]
 45%|████▌     | 4543/10000 [31:24<16:14,  5.60it/s]
 45%|████▌     | 4544/10000 [31:24<16:20,  5.57it/s]
 45%|████▌     | 4545/10000 [31:24<16:22,  5.55it/s]
 45%|████▌     | 4546/10000 [31:24<16:16,  5.59it/s]
 45%|████▌     | 4547/10000 [31:24<16:06,  5.64it/s]
 45%|████▌     | 4548/10000 [31:25<16:00,  5.67it/s]
 45%|████▌     | 4549/10000 [31:25<15:59,  5.68it/s]
 46%|████▌     | 4550/10000 [31:25<15:56,  5.70it/s]
                                                    
{'loss': 0.085, 'grad_norm': 0.7977578639984131, 'learning_rate': 6.148728903632081e-05}

 46%|████▌     | 4550/10000 [31:25<15:56,  5.70it/s]
 46%|████▌     | 4551/10000 [31:25<16:10,  5.62it/s]
 46%|████▌     | 4552/10000 [31:25<16:06,  5.64it/s]
 46%|████▌     | 4553/10000 [31:26<16:03,  5.65it/s]
 46%|████▌     | 4554/10000 [31:26<15:55,  5.70it/s]
 46%|████▌     | 4555/10000 [31:26<15:53,  5.71it/s]
 46%|████▌     | 4556/10000 [31:26<15:51,  5.72it/s]
 46%|████▌     | 4557/10000 [31:26<15:44,  5.76it/s]
 46%|████▌     | 4558/10000 [31:26<15:38,  5.80it/s]
 46%|████▌     | 4559/10000 [31:27<15:45,  5.76it/s]
 46%|████▌     | 4560/10000 [31:27<15:40,  5.78it/s]
                                                    
{'loss': 0.0919, 'grad_norm': 0.7852060198783875, 'learning_rate': 6.132630245644921e-05}

 46%|████▌     | 4560/10000 [31:27<15:40,  5.78it/s]
 46%|████▌     | 4561/10000 [31:27<15:58,  5.67it/s]
 46%|████▌     | 4562/10000 [31:27<15:56,  5.69it/s]
 46%|████▌     | 4563/10000 [31:27<15:50,  5.72it/s]
 46%|████▌     | 4564/10000 [31:27<15:42,  5.77it/s]
 46%|████▌     | 4565/10000 [31:28<15:46,  5.74it/s]Rank 0, Worker 5: Wait for shard 14 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 46%|████▌     | 4566/10000 [31:28<16:10,  5.60it/s]
 46%|████▌     | 4567/10000 [31:28<16:41,  5.42it/s]
 46%|████▌     | 4568/10000 [31:28<17:58,  5.04it/s]
 46%|████▌     | 4569/10000 [31:28<18:29,  4.90it/s]
 46%|████▌     | 4570/10000 [31:29<19:13,  4.71it/s]
                                                    
{'loss': 0.0878, 'grad_norm': 0.7635006308555603, 'learning_rate': 6.116519201394857e-05}

 46%|████▌     | 4570/10000 [31:29<19:13,  4.71it/s]
 46%|████▌     | 4571/10000 [31:29<20:50,  4.34it/s]
 46%|████▌     | 4572/10000 [31:29<20:36,  4.39it/s]
 46%|████▌     | 4573/10000 [31:29<20:20,  4.45it/s]
 46%|████▌     | 4574/10000 [31:30<19:58,  4.53it/s]
 46%|████▌     | 4575/10000 [31:30<19:42,  4.59it/s]
 46%|████▌     | 4576/10000 [31:30<19:29,  4.64it/s]
 46%|████▌     | 4577/10000 [31:30<19:38,  4.60it/s]
 46%|████▌     | 4578/10000 [31:30<19:57,  4.53it/s]
 46%|████▌     | 4579/10000 [31:31<19:32,  4.62it/s]
 46%|████▌     | 4580/10000 [31:31<19:11,  4.71it/s]
                                                    
{'loss': 0.0945, 'grad_norm': 0.858343243598938, 'learning_rate': 6.10039594706969e-05}

 46%|████▌     | 4580/10000 [31:31<19:11,  4.71it/s]
 46%|████▌     | 4581/10000 [31:31<19:01,  4.75it/s]
 46%|████▌     | 4582/10000 [31:31<18:18,  4.93it/s]
 46%|████▌     | 4583/10000 [31:31<18:11,  4.96it/s]
 46%|████▌     | 4584/10000 [31:32<18:25,  4.90it/s]
 46%|████▌     | 4585/10000 [31:32<18:57,  4.76it/s]
 46%|████▌     | 4586/10000 [31:32<18:50,  4.79it/s]
 46%|████▌     | 4587/10000 [31:32<19:02,  4.74it/s]Rank 0, Worker 3: Wait for shard 0 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 46%|████▌     | 4588/10000 [31:33<19:15,  4.68it/s]
 46%|████▌     | 4589/10000 [31:33<19:07,  4.72it/s]
 46%|████▌     | 4590/10000 [31:33<18:57,  4.75it/s]
                                                    
{'loss': 0.0955, 'grad_norm': 0.7552047967910767, 'learning_rate': 6.084260658990744e-05}

 46%|████▌     | 4590/10000 [31:33<18:57,  4.75it/s]
 46%|████▌     | 4591/10000 [31:33<19:00,  4.74it/s]
 46%|████▌     | 4592/10000 [31:33<18:43,  4.81it/s]
 46%|████▌     | 4593/10000 [31:34<18:35,  4.85it/s]
 46%|████▌     | 4594/10000 [31:34<18:27,  4.88it/s]
 46%|████▌     | 4595/10000 [31:34<18:42,  4.81it/s]
 46%|████▌     | 4596/10000 [31:34<19:07,  4.71it/s]
 46%|████▌     | 4597/10000 [31:34<19:01,  4.73it/s]
 46%|████▌     | 4598/10000 [31:35<19:29,  4.62it/s]
 46%|████▌     | 4599/10000 [31:35<20:30,  4.39it/s]
 46%|████▌     | 4600/10000 [31:35<20:55,  4.30it/s]
                                                    
{'loss': 0.1092, 'grad_norm': 0.81109219789505, 'learning_rate': 6.068113513610943e-05}

 46%|████▌     | 4600/10000 [31:35<20:55,  4.30it/s]
 46%|████▌     | 4601/10000 [31:35<21:25,  4.20it/s]
 46%|████▌     | 4602/10000 [31:36<21:20,  4.22it/s]
 46%|████▌     | 4603/10000 [31:36<20:51,  4.31it/s]
 46%|████▌     | 4604/10000 [31:36<20:56,  4.30it/s]
 46%|████▌     | 4605/10000 [31:36<21:26,  4.19it/s]
 46%|████▌     | 4606/10000 [31:37<22:09,  4.06it/s]
 46%|████▌     | 4607/10000 [31:37<22:55,  3.92it/s]
 46%|████▌     | 4608/10000 [31:37<22:54,  3.92it/s]
 46%|████▌     | 4609/10000 [31:37<22:18,  4.03it/s]
 46%|████▌     | 4610/10000 [31:38<21:22,  4.20it/s]
                                                    
{'loss': 0.1033, 'grad_norm': 0.9269545674324036, 'learning_rate': 6.0519546875128876e-05}

 46%|████▌     | 4610/10000 [31:38<21:22,  4.20it/s]
 46%|████▌     | 4611/10000 [31:38<20:17,  4.43it/s]
 46%|████▌     | 4612/10000 [31:38<19:43,  4.55it/s]
 46%|████▌     | 4613/10000 [31:38<19:20,  4.64it/s]
 46%|████▌     | 4614/10000 [31:38<19:22,  4.63it/s]
 46%|████▌     | 4615/10000 [31:39<19:36,  4.58it/s]
 46%|████▌     | 4616/10000 [31:39<20:05,  4.47it/s]
 46%|████▌     | 4617/10000 [31:39<20:09,  4.45it/s]
 46%|████▌     | 4618/10000 [31:39<19:45,  4.54it/s]
 46%|████▌     | 4619/10000 [31:39<19:10,  4.68it/s]
 46%|████▌     | 4620/10000 [31:40<18:46,  4.78it/s]
                                                    
{'loss': 0.0886, 'grad_norm': 0.7793946266174316, 'learning_rate': 6.035784357406906e-05}

 46%|████▌     | 4620/10000 [31:40<18:46,  4.78it/s]
 46%|████▌     | 4621/10000 [31:40<19:29,  4.60it/s]
 46%|████▌     | 4622/10000 [31:40<19:37,  4.57it/s]
 46%|████▌     | 4623/10000 [31:40<19:34,  4.58it/s]
 46%|████▌     | 4624/10000 [31:41<19:37,  4.57it/s]
 46%|████▋     | 4625/10000 [31:41<19:29,  4.59it/s]
 46%|████▋     | 4626/10000 [31:41<19:02,  4.70it/s]
 46%|████▋     | 4627/10000 [31:41<18:36,  4.81it/s]
 46%|████▋     | 4628/10000 [31:41<17:57,  4.99it/s]
 46%|████▋     | 4629/10000 [31:42<17:54,  5.00it/s]
 46%|████▋     | 4630/10000 [31:42<18:08,  4.93it/s]
                                                    
{'loss': 0.0951, 'grad_norm': 0.8939802646636963, 'learning_rate': 6.01960270012914e-05}

 46%|████▋     | 4630/10000 [31:42<18:08,  4.93it/s]
 46%|████▋     | 4631/10000 [31:42<18:44,  4.77it/s]
 46%|████▋     | 4632/10000 [31:42<18:39,  4.79it/s]
 46%|████▋     | 4633/10000 [31:42<18:48,  4.76it/s]
 46%|████▋     | 4634/10000 [31:43<18:28,  4.84it/s]
 46%|████▋     | 4635/10000 [31:43<18:08,  4.93it/s]
 46%|████▋     | 4636/10000 [31:43<17:58,  4.97it/s]
 46%|████▋     | 4637/10000 [31:43<18:00,  4.96it/s]
 46%|████▋     | 4638/10000 [31:43<18:00,  4.96it/s]
 46%|████▋     | 4639/10000 [31:44<18:06,  4.93it/s]
 46%|████▋     | 4640/10000 [31:44<18:12,  4.91it/s]
                                                    
{'loss': 0.0902, 'grad_norm': 0.7566599249839783, 'learning_rate': 6.003409892639599e-05}

 46%|████▋     | 4640/10000 [31:44<18:12,  4.91it/s]
 46%|████▋     | 4641/10000 [31:44<18:40,  4.78it/s]
 46%|████▋     | 4642/10000 [31:44<18:31,  4.82it/s]
 46%|████▋     | 4643/10000 [31:44<18:15,  4.89it/s]
 46%|████▋     | 4644/10000 [31:45<18:15,  4.89it/s]
 46%|████▋     | 4645/10000 [31:45<18:03,  4.94it/s]
 46%|████▋     | 4646/10000 [31:45<17:48,  5.01it/s]
 46%|████▋     | 4647/10000 [31:45<17:57,  4.97it/s]
 46%|████▋     | 4648/10000 [31:45<18:37,  4.79it/s]
 46%|████▋     | 4649/10000 [31:46<19:13,  4.64it/s]
 46%|████▋     | 4650/10000 [31:46<19:09,  4.66it/s]
                                                    
{'loss': 0.0939, 'grad_norm': 0.9234247207641602, 'learning_rate': 5.9872061120202336e-05}

 46%|████▋     | 4650/10000 [31:46<19:09,  4.66it/s]
 47%|████▋     | 4651/10000 [31:46<19:44,  4.52it/s]
 47%|████▋     | 4652/10000 [31:46<19:41,  4.53it/s]
 47%|████▋     | 4653/10000 [31:47<19:33,  4.56it/s]
 47%|████▋     | 4654/10000 [31:47<19:08,  4.66it/s]
 47%|████▋     | 4655/10000 [31:47<18:37,  4.78it/s]
 47%|████▋     | 4656/10000 [31:47<18:17,  4.87it/s]
 47%|████▋     | 4657/10000 [31:47<18:13,  4.89it/s]
 47%|████▋     | 4658/10000 [31:48<18:13,  4.88it/s]
 47%|████▋     | 4659/10000 [31:48<18:17,  4.87it/s]
 47%|████▋     | 4660/10000 [31:48<18:43,  4.75it/s]
                                                    
{'loss': 0.1075, 'grad_norm': 1.1718734502792358, 'learning_rate': 5.9709915354729914e-05}

 47%|████▋     | 4660/10000 [31:48<18:43,  4.75it/s]
 47%|████▋     | 4661/10000 [31:48<19:03,  4.67it/s]
 47%|████▋     | 4662/10000 [31:48<18:57,  4.69it/s]
 47%|████▋     | 4663/10000 [31:49<18:41,  4.76it/s]
 47%|████▋     | 4664/10000 [31:49<18:15,  4.87it/s]Rank 0, Worker 2: Wait for shard 74 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 47%|████▋     | 4665/10000 [31:49<18:12,  4.88it/s]
 47%|████▋     | 4666/10000 [31:49<18:00,  4.93it/s]
 47%|████▋     | 4667/10000 [31:49<18:03,  4.92it/s]
 47%|████▋     | 4668/10000 [31:50<18:17,  4.86it/s]
 47%|████▋     | 4669/10000 [31:50<18:10,  4.89it/s]
 47%|████▋     | 4670/10000 [31:50<18:54,  4.70it/s]
                                                    
{'loss': 0.1005, 'grad_norm': 0.7688272595405579, 'learning_rate': 5.9547663403178824e-05}

 47%|████▋     | 4670/10000 [31:50<18:54,  4.70it/s]
 47%|████▋     | 4671/10000 [31:50<20:00,  4.44it/s]
 47%|████▋     | 4672/10000 [31:51<19:53,  4.46it/s]
 47%|████▋     | 4673/10000 [31:51<20:12,  4.39it/s]
 47%|████▋     | 4674/10000 [31:51<19:59,  4.44it/s]
 47%|████▋     | 4675/10000 [31:51<19:30,  4.55it/s]
 47%|████▋     | 4676/10000 [31:51<18:56,  4.68it/s]
 47%|████▋     | 4677/10000 [31:52<18:49,  4.71it/s]
 47%|████▋     | 4678/10000 [31:52<18:44,  4.73it/s]
 47%|████▋     | 4679/10000 [31:52<19:13,  4.61it/s]
 47%|████▋     | 4680/10000 [31:52<19:45,  4.49it/s]
                                                    
{'loss': 0.1146, 'grad_norm': 0.9440938234329224, 'learning_rate': 5.9385307039910445e-05}

 47%|████▋     | 4680/10000 [31:52<19:45,  4.49it/s]
 47%|████▋     | 4681/10000 [31:53<20:57,  4.23it/s]
 47%|████▋     | 4682/10000 [31:53<22:05,  4.01it/s]
 47%|████▋     | 4683/10000 [31:53<20:45,  4.27it/s]
 47%|████▋     | 4684/10000 [31:53<19:34,  4.53it/s]
 47%|████▋     | 4685/10000 [31:54<19:48,  4.47it/s]
 47%|████▋     | 4686/10000 [31:54<19:26,  4.56it/s]
 47%|████▋     | 4687/10000 [31:54<19:42,  4.49it/s]
 47%|████▋     | 4688/10000 [31:54<20:32,  4.31it/s]
 47%|████▋     | 4689/10000 [31:54<20:31,  4.31it/s]
 47%|████▋     | 4690/10000 [31:55<20:26,  4.33it/s]
                                                    
{'loss': 0.1043, 'grad_norm': 0.798103928565979, 'learning_rate': 5.922284804042792e-05}

 47%|████▋     | 4690/10000 [31:55<20:26,  4.33it/s]
 47%|████▋     | 4691/10000 [31:55<20:11,  4.38it/s]
 47%|████▋     | 4692/10000 [31:55<19:29,  4.54it/s]
 47%|████▋     | 4693/10000 [31:55<18:58,  4.66it/s]
 47%|████▋     | 4694/10000 [31:55<18:35,  4.76it/s]
 47%|████▋     | 4695/10000 [31:56<18:32,  4.77it/s]
 47%|████▋     | 4696/10000 [31:56<18:38,  4.74it/s]
 47%|████▋     | 4697/10000 [31:56<18:48,  4.70it/s]
 47%|████▋     | 4698/10000 [31:56<18:45,  4.71it/s]
 47%|████▋     | 4699/10000 [31:57<18:54,  4.67it/s]
 47%|████▋     | 4700/10000 [31:57<18:40,  4.73it/s]
                                                    
{'loss': 0.1129, 'grad_norm': 0.9380629658699036, 'learning_rate': 5.906028818135687e-05}

 47%|████▋     | 4700/10000 [31:57<18:40,  4.73it/s]
 47%|████▋     | 4701/10000 [31:57<19:12,  4.60it/s]
 47%|████▋     | 4702/10000 [31:57<19:13,  4.59it/s]
 47%|████▋     | 4703/10000 [31:57<19:08,  4.61it/s]
 47%|████▋     | 4704/10000 [31:58<19:33,  4.51it/s]
 47%|████▋     | 4705/10000 [31:58<19:22,  4.55it/s]
 47%|████▋     | 4706/10000 [31:58<18:54,  4.67it/s]
 47%|████▋     | 4707/10000 [31:58<18:42,  4.72it/s]
 47%|████▋     | 4708/10000 [31:59<19:01,  4.64it/s]
 47%|████▋     | 4709/10000 [31:59<19:01,  4.63it/s]
 47%|████▋     | 4710/10000 [31:59<19:18,  4.56it/s]
                                                    
{'loss': 0.1078, 'grad_norm': 0.8713814616203308, 'learning_rate': 5.889762924042585e-05}

 47%|████▋     | 4710/10000 [31:59<19:18,  4.56it/s]
 47%|████▋     | 4711/10000 [31:59<20:14,  4.36it/s]
 47%|████▋     | 4712/10000 [31:59<19:50,  4.44it/s]
 47%|████▋     | 4713/10000 [32:00<19:51,  4.44it/s]
 47%|████▋     | 4714/10000 [32:00<19:18,  4.56it/s]
 47%|████▋     | 4715/10000 [32:00<18:55,  4.66it/s]
 47%|████▋     | 4716/10000 [32:00<18:56,  4.65it/s]
 47%|████▋     | 4717/10000 [32:00<18:46,  4.69it/s]
 47%|████▋     | 4718/10000 [32:01<18:35,  4.74it/s]
 47%|████▋     | 4719/10000 [32:01<18:50,  4.67it/s]
 47%|████▋     | 4720/10000 [32:01<18:46,  4.69it/s]
                                                    
{'loss': 0.095, 'grad_norm': 0.8485936522483826, 'learning_rate': 5.873487299644699e-05}

 47%|████▋     | 4720/10000 [32:01<18:46,  4.69it/s]
 47%|████▋     | 4721/10000 [32:01<18:51,  4.66it/s]
 47%|████▋     | 4722/10000 [32:02<18:53,  4.66it/s]
 47%|████▋     | 4723/10000 [32:02<18:32,  4.74it/s]
 47%|████▋     | 4724/10000 [32:02<18:07,  4.85it/s]
 47%|████▋     | 4725/10000 [32:02<17:48,  4.94it/s]
 47%|████▋     | 4726/10000 [32:02<17:44,  4.95it/s]
 47%|████▋     | 4727/10000 [32:03<17:35,  5.00it/s]
 47%|████▋     | 4728/10000 [32:03<17:36,  4.99it/s]
 47%|████▋     | 4729/10000 [32:03<17:31,  5.01it/s]
 47%|████▋     | 4730/10000 [32:03<17:29,  5.02it/s]
                                                    
{'loss': 0.0882, 'grad_norm': 0.897530734539032, 'learning_rate': 5.857202122929649e-05}

 47%|████▋     | 4730/10000 [32:03<17:29,  5.02it/s]
 47%|████▋     | 4731/10000 [32:03<17:30,  5.01it/s]
 47%|████▋     | 4732/10000 [32:04<17:11,  5.11it/s]
 47%|████▋     | 4733/10000 [32:04<16:58,  5.17it/s]
 47%|████▋     | 4734/10000 [32:04<17:09,  5.11it/s]Rank 0, Worker 0: Wait for shard 18 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 47%|████▋     | 4735/10000 [32:04<17:34,  4.99it/s]
 47%|████▋     | 4736/10000 [32:04<17:41,  4.96it/s]
 47%|████▋     | 4737/10000 [32:05<17:38,  4.97it/s]
 47%|████▋     | 4738/10000 [32:05<17:36,  4.98it/s]
 47%|████▋     | 4739/10000 [32:05<17:21,  5.05it/s]
 47%|████▋     | 4740/10000 [32:05<17:17,  5.07it/s]
                                                    
{'loss': 0.0938, 'grad_norm': 0.7317162752151489, 'learning_rate': 5.840907571989518e-05}

 47%|████▋     | 4740/10000 [32:05<17:17,  5.07it/s]
 47%|████▋     | 4741/10000 [32:05<17:23,  5.04it/s]
 47%|████▋     | 4742/10000 [32:05<16:56,  5.17it/s]
 47%|████▋     | 4743/10000 [32:06<16:31,  5.30it/s]
 47%|████▋     | 4744/10000 [32:06<16:14,  5.39it/s]
 47%|████▋     | 4745/10000 [32:06<16:06,  5.44it/s]
 47%|████▋     | 4746/10000 [32:06<15:54,  5.51it/s]
 47%|████▋     | 4747/10000 [32:06<16:11,  5.41it/s]
 47%|████▋     | 4748/10000 [32:07<16:23,  5.34it/s]
 47%|████▋     | 4749/10000 [32:07<16:24,  5.33it/s]
 48%|████▊     | 4750/10000 [32:07<16:23,  5.34it/s]Rank 0, Worker 0: Caching shard...Rank 0, Worker 1: Caching shard...Rank 0, Worker 2: Caching shard...


Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.08 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.35 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 16.59 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 16.81 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 16.94 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 16.96 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 4750: eval/loss = 0.1029 (35s)

                                                    
{'loss': 0.1161, 'grad_norm': 0.7318452000617981, 'learning_rate': 5.824603825018904e-05}

 48%|████▊     | 4750/10000 [32:42<16:23,  5.34it/s]
 48%|████▊     | 4751/10000 [32:42<15:21:55, 10.54s/it]
 48%|████▊     | 4752/10000 [32:42<10:49:44,  7.43s/it]
 48%|████▊     | 4753/10000 [32:42<7:39:14,  5.25s/it] 
 48%|████▊     | 4754/10000 [32:42<5:26:10,  3.73s/it]
 48%|████▊     | 4755/10000 [32:42<3:53:04,  2.67s/it]
 48%|████▊     | 4756/10000 [32:43<2:47:40,  1.92s/it]
 48%|████▊     | 4757/10000 [32:43<2:01:59,  1.40s/it]
 48%|████▊     | 4758/10000 [32:43<1:30:15,  1.03s/it]
 48%|████▊     | 4759/10000 [32:43<1:07:39,  1.29it/s]
 48%|████▊     | 4760/10000 [32:43<52:14,  1.67it/s]  
                                                    
{'loss': 0.084, 'grad_norm': 0.9321379661560059, 'learning_rate': 5.808291060312975e-05}

 48%|████▊     | 4760/10000 [32:43<52:14,  1.67it/s]
 48%|████▊     | 4761/10000 [32:43<41:43,  2.09it/s]
 48%|████▊     | 4762/10000 [32:44<34:13,  2.55it/s]
 48%|████▊     | 4763/10000 [32:44<28:37,  3.05it/s]
 48%|████▊     | 4764/10000 [32:44<25:30,  3.42it/s]
 48%|████▊     | 4765/10000 [32:44<23:04,  3.78it/s]
 48%|████▊     | 4766/10000 [32:44<21:29,  4.06it/s]
 48%|████▊     | 4767/10000 [32:45<20:46,  4.20it/s]
 48%|████▊     | 4768/10000 [32:45<20:02,  4.35it/s]
 48%|████▊     | 4769/10000 [32:45<19:48,  4.40it/s]
 48%|████▊     | 4770/10000 [32:45<18:58,  4.59it/s]
                                                    
{'loss': 0.1211, 'grad_norm': 0.9103684425354004, 'learning_rate': 5.7919694562655083e-05}

 48%|████▊     | 4770/10000 [32:45<18:58,  4.59it/s]
 48%|████▊     | 4771/10000 [32:45<18:36,  4.68it/s]
 48%|████▊     | 4772/10000 [32:46<17:46,  4.90it/s]
 48%|████▊     | 4773/10000 [32:46<17:11,  5.07it/s]
 48%|████▊     | 4774/10000 [32:46<16:42,  5.21it/s]
 48%|████▊     | 4775/10000 [32:46<16:41,  5.22it/s]
 48%|████▊     | 4776/10000 [32:46<16:34,  5.25it/s]
 48%|████▊     | 4777/10000 [32:47<16:43,  5.20it/s]
 48%|████▊     | 4778/10000 [32:47<16:48,  5.18it/s]
 48%|████▊     | 4779/10000 [32:47<16:47,  5.18it/s]
 48%|████▊     | 4780/10000 [32:47<16:56,  5.14it/s]
                                                    
{'loss': 0.0972, 'grad_norm': 0.8509300947189331, 'learning_rate': 5.775639191366954e-05}

 48%|████▊     | 4780/10000 [32:47<16:56,  5.14it/s]
 48%|████▊     | 4781/10000 [32:47<17:30,  4.97it/s]
 48%|████▊     | 4782/10000 [32:48<17:22,  5.01it/s]
 48%|████▊     | 4783/10000 [32:48<16:50,  5.16it/s]
 48%|████▊     | 4784/10000 [32:48<16:40,  5.21it/s]
 48%|████▊     | 4785/10000 [32:48<16:22,  5.31it/s]
 48%|████▊     | 4786/10000 [32:48<16:16,  5.34it/s]
 48%|████▊     | 4787/10000 [32:49<16:21,  5.31it/s]
 48%|████▊     | 4788/10000 [32:49<16:24,  5.29it/s]
 48%|████▊     | 4789/10000 [32:49<16:37,  5.22it/s]
 48%|████▊     | 4790/10000 [32:49<16:44,  5.19it/s]
                                                    
{'loss': 0.1109, 'grad_norm': 1.0333175659179688, 'learning_rate': 5.75930044420247e-05}

 48%|████▊     | 4790/10000 [32:49<16:44,  5.19it/s]
 48%|████▊     | 4791/10000 [32:49<16:57,  5.12it/s]
 48%|████▊     | 4792/10000 [32:50<16:54,  5.14it/s]
 48%|████▊     | 4793/10000 [32:50<16:50,  5.15it/s]
 48%|████▊     | 4794/10000 [32:50<16:31,  5.25it/s]
 48%|████▊     | 4795/10000 [32:50<16:10,  5.36it/s]
 48%|████▊     | 4796/10000 [32:50<16:33,  5.24it/s]
 48%|████▊     | 4797/10000 [32:50<16:47,  5.16it/s]
 48%|████▊     | 4798/10000 [32:51<17:20,  5.00it/s]
 48%|████▊     | 4799/10000 [32:51<18:09,  4.77it/s]
 48%|████▊     | 4800/10000 [32:51<18:31,  4.68it/s]
                                                    
{'loss': 0.0978, 'grad_norm': 0.7124407291412354, 'learning_rate': 5.74295339344998e-05}

 48%|████▊     | 4800/10000 [32:51<18:31,  4.68it/s]
 48%|████▊     | 4801/10000 [32:51<18:53,  4.59it/s]
 48%|████▊     | 4802/10000 [32:52<18:48,  4.61it/s]
 48%|████▊     | 4803/10000 [32:52<18:36,  4.65it/s]
 48%|████▊     | 4804/10000 [32:52<18:21,  4.72it/s]
 48%|████▊     | 4805/10000 [32:52<18:24,  4.70it/s]
 48%|████▊     | 4806/10000 [32:52<18:29,  4.68it/s]
 48%|████▊     | 4807/10000 [32:53<19:11,  4.51it/s]
 48%|████▊     | 4808/10000 [32:53<19:17,  4.49it/s]
 48%|████▊     | 4809/10000 [32:53<19:44,  4.38it/s]
 48%|████▊     | 4810/10000 [32:53<19:45,  4.38it/s]
                                                    
{'loss': 0.0918, 'grad_norm': 0.7871521711349487, 'learning_rate': 5.726598217878211e-05}

 48%|████▊     | 4810/10000 [32:53<19:45,  4.38it/s]
 48%|████▊     | 4811/10000 [32:54<19:35,  4.41it/s]
 48%|████▊     | 4812/10000 [32:54<18:40,  4.63it/s]
 48%|████▊     | 4813/10000 [32:54<17:52,  4.84it/s]
 48%|████▊     | 4814/10000 [32:54<17:11,  5.03it/s]
 48%|████▊     | 4815/10000 [32:54<17:08,  5.04it/s]
 48%|████▊     | 4816/10000 [32:55<17:13,  5.02it/s]
 48%|████▊     | 4817/10000 [32:55<17:27,  4.95it/s]
 48%|████▊     | 4818/10000 [32:55<18:00,  4.80it/s]
 48%|████▊     | 4819/10000 [32:55<17:49,  4.84it/s]
 48%|████▊     | 4820/10000 [32:55<17:38,  4.89it/s]
                                                    
{'loss': 0.106, 'grad_norm': 0.8016433119773865, 'learning_rate': 5.71023509634474e-05}

 48%|████▊     | 4820/10000 [32:55<17:38,  4.89it/s]
 48%|████▊     | 4821/10000 [32:56<17:47,  4.85it/s]
 48%|████▊     | 4822/10000 [32:56<17:13,  5.01it/s]
 48%|████▊     | 4823/10000 [32:56<16:43,  5.16it/s]
 48%|████▊     | 4824/10000 [32:56<16:40,  5.17it/s]
 48%|████▊     | 4825/10000 [32:56<17:07,  5.04it/s]
 48%|████▊     | 4826/10000 [32:57<17:49,  4.84it/s]
 48%|████▊     | 4827/10000 [32:57<18:59,  4.54it/s]
 48%|████▊     | 4828/10000 [32:57<18:53,  4.56it/s]
 48%|████▊     | 4829/10000 [32:57<18:14,  4.72it/s]
 48%|████▊     | 4830/10000 [32:57<17:42,  4.86it/s]
                                                    
{'loss': 0.1217, 'grad_norm': 0.9009758234024048, 'learning_rate': 5.693864207794049e-05}

 48%|████▊     | 4830/10000 [32:57<17:42,  4.86it/s]
 48%|████▊     | 4831/10000 [32:58<17:35,  4.90it/s]
 48%|████▊     | 4832/10000 [32:58<17:40,  4.87it/s]
 48%|████▊     | 4833/10000 [32:58<17:40,  4.87it/s]
 48%|████▊     | 4834/10000 [32:58<17:34,  4.90it/s]
 48%|████▊     | 4835/10000 [32:58<17:08,  5.02it/s]
 48%|████▊     | 4836/10000 [32:59<16:59,  5.07it/s]
 48%|████▊     | 4837/10000 [32:59<16:57,  5.07it/s]
 48%|████▊     | 4838/10000 [32:59<17:07,  5.03it/s]
 48%|████▊     | 4839/10000 [32:59<17:14,  4.99it/s]
 48%|████▊     | 4840/10000 [32:59<17:05,  5.03it/s]
                                                    
{'loss': 0.1014, 'grad_norm': 1.020508050918579, 'learning_rate': 5.677485731255545e-05}

 48%|████▊     | 4840/10000 [32:59<17:05,  5.03it/s]
 48%|████▊     | 4841/10000 [33:00<17:05,  5.03it/s]
 48%|████▊     | 4842/10000 [33:00<16:39,  5.16it/s]
 48%|████▊     | 4843/10000 [33:00<16:39,  5.16it/s]
 48%|████▊     | 4844/10000 [33:00<16:41,  5.15it/s]
 48%|████▊     | 4845/10000 [33:00<17:22,  4.95it/s]
 48%|████▊     | 4846/10000 [33:01<17:43,  4.85it/s]
 48%|████▊     | 4847/10000 [33:01<17:21,  4.95it/s]
 48%|████▊     | 4848/10000 [33:01<16:55,  5.07it/s]
 48%|████▊     | 4849/10000 [33:01<16:20,  5.26it/s]
 48%|████▊     | 4850/10000 [33:01<16:02,  5.35it/s]
                                                    
{'loss': 0.1104, 'grad_norm': 0.759361982345581, 'learning_rate': 5.6610998458416296e-05}

 48%|████▊     | 4850/10000 [33:01<16:02,  5.35it/s]
 49%|████▊     | 4851/10000 [33:02<16:17,  5.27it/s]
 49%|████▊     | 4852/10000 [33:02<16:27,  5.21it/s]
 49%|████▊     | 4853/10000 [33:02<16:48,  5.11it/s]
 49%|████▊     | 4854/10000 [33:02<16:59,  5.05it/s]
 49%|████▊     | 4855/10000 [33:02<17:28,  4.91it/s]
 49%|████▊     | 4856/10000 [33:03<17:18,  4.96it/s]
 49%|████▊     | 4857/10000 [33:03<16:57,  5.05it/s]
 49%|████▊     | 4858/10000 [33:03<16:24,  5.22it/s]
 49%|████▊     | 4859/10000 [33:03<16:04,  5.33it/s]
 49%|████▊     | 4860/10000 [33:03<15:55,  5.38it/s]
                                                    
{'loss': 0.0984, 'grad_norm': 0.9707552790641785, 'learning_rate': 5.644706730745716e-05}

 49%|████▊     | 4860/10000 [33:03<15:55,  5.38it/s]
 49%|████▊     | 4861/10000 [33:04<16:31,  5.18it/s]
 49%|████▊     | 4862/10000 [33:04<17:04,  5.01it/s]
 49%|████▊     | 4863/10000 [33:04<17:42,  4.84it/s]
 49%|████▊     | 4864/10000 [33:04<18:04,  4.73it/s]
 49%|████▊     | 4865/10000 [33:04<17:55,  4.77it/s]
 49%|████▊     | 4866/10000 [33:05<17:23,  4.92it/s]
 49%|████▊     | 4867/10000 [33:05<16:57,  5.04it/s]
 49%|████▊     | 4868/10000 [33:05<16:33,  5.17it/s]
 49%|████▊     | 4869/10000 [33:05<16:27,  5.19it/s]
 49%|████▊     | 4870/10000 [33:05<16:42,  5.12it/s]
                                                    
{'loss': 0.1031, 'grad_norm': 0.7132483720779419, 'learning_rate': 5.628306565240287e-05}

 49%|████▊     | 4870/10000 [33:05<16:42,  5.12it/s]
 49%|████▊     | 4871/10000 [33:06<17:08,  4.99it/s]
 49%|████▊     | 4872/10000 [33:06<17:23,  4.91it/s]
 49%|████▊     | 4873/10000 [33:06<17:19,  4.93it/s]
 49%|████▊     | 4874/10000 [33:06<17:29,  4.88it/s]
 49%|████▉     | 4875/10000 [33:06<17:35,  4.86it/s]
 49%|████▉     | 4876/10000 [33:07<18:04,  4.72it/s]
 49%|████▉     | 4877/10000 [33:07<17:28,  4.88it/s]
 49%|████▉     | 4878/10000 [33:07<17:41,  4.83it/s]
 49%|████▉     | 4879/10000 [33:07<17:30,  4.88it/s]
 49%|████▉     | 4880/10000 [33:07<17:24,  4.90it/s]
                                                    
{'loss': 0.1108, 'grad_norm': 1.0457226037979126, 'learning_rate': 5.611899528674923e-05}

 49%|████▉     | 4880/10000 [33:07<17:24,  4.90it/s]
 49%|████▉     | 4881/10000 [33:08<17:37,  4.84it/s]
 49%|████▉     | 4882/10000 [33:08<17:59,  4.74it/s]
 49%|████▉     | 4883/10000 [33:08<18:14,  4.68it/s]
 49%|████▉     | 4884/10000 [33:08<18:13,  4.68it/s]
 49%|████▉     | 4885/10000 [33:08<18:06,  4.71it/s]
 49%|████▉     | 4886/10000 [33:09<17:11,  4.96it/s]
 49%|████▉     | 4887/10000 [33:09<16:29,  5.17it/s]
 49%|████▉     | 4888/10000 [33:09<16:03,  5.31it/s]
 49%|████▉     | 4889/10000 [33:09<15:51,  5.37it/s]
 49%|████▉     | 4890/10000 [33:09<16:04,  5.30it/s]
                                                    
{'loss': 0.1057, 'grad_norm': 0.8774409890174866, 'learning_rate': 5.595485800474349e-05}

 49%|████▉     | 4890/10000 [33:09<16:04,  5.30it/s]
 49%|████▉     | 4891/10000 [33:10<16:41,  5.10it/s]
 49%|████▉     | 4892/10000 [33:10<16:58,  5.01it/s]
 49%|████▉     | 4893/10000 [33:10<16:58,  5.01it/s]
 49%|████▉     | 4894/10000 [33:10<16:43,  5.09it/s]
 49%|████▉     | 4895/10000 [33:10<16:16,  5.23it/s]
 49%|████▉     | 4896/10000 [33:11<15:52,  5.36it/s]
 49%|████▉     | 4897/10000 [33:11<15:53,  5.35it/s]
 49%|████▉     | 4898/10000 [33:11<16:07,  5.27it/s]
 49%|████▉     | 4899/10000 [33:11<16:38,  5.11it/s]
 49%|████▉     | 4900/10000 [33:11<17:01,  4.99it/s]
                                                    
{'loss': 0.0786, 'grad_norm': 0.8555901050567627, 'learning_rate': 5.579065560136467e-05}

 49%|████▉     | 4900/10000 [33:11<17:01,  4.99it/s]
 49%|████▉     | 4901/10000 [33:12<17:17,  4.91it/s]
 49%|████▉     | 4902/10000 [33:12<16:56,  5.01it/s]
 49%|████▉     | 4903/10000 [33:12<16:40,  5.10it/s]
 49%|████▉     | 4904/10000 [33:12<16:15,  5.22it/s]
 49%|████▉     | 4905/10000 [33:12<15:55,  5.33it/s]
 49%|████▉     | 4906/10000 [33:12<15:50,  5.36it/s]
 49%|████▉     | 4907/10000 [33:13<16:22,  5.18it/s]
 49%|████▉     | 4908/10000 [33:13<17:41,  4.80it/s]
 49%|████▉     | 4909/10000 [33:13<17:47,  4.77it/s]
 49%|████▉     | 4910/10000 [33:13<17:48,  4.76it/s]
                                                    
{'loss': 0.1132, 'grad_norm': 0.9465853571891785, 'learning_rate': 5.562638987230392e-05}

 49%|████▉     | 4910/10000 [33:13<17:48,  4.76it/s]
 49%|████▉     | 4911/10000 [33:14<17:27,  4.86it/s]
 49%|████▉     | 4912/10000 [33:14<17:57,  4.72it/s]
 49%|████▉     | 4913/10000 [33:14<17:53,  4.74it/s]
 49%|████▉     | 4914/10000 [33:14<18:01,  4.70it/s]
 49%|████▉     | 4915/10000 [33:14<17:45,  4.77it/s]
 49%|████▉     | 4916/10000 [33:15<17:28,  4.85it/s]
 49%|████▉     | 4917/10000 [33:15<17:17,  4.90it/s]
 49%|████▉     | 4918/10000 [33:15<17:06,  4.95it/s]
 49%|████▉     | 4919/10000 [33:15<16:57,  4.99it/s]
 49%|████▉     | 4920/10000 [33:15<16:43,  5.06it/s]
                                                    
{'loss': 0.1047, 'grad_norm': 0.8607863783836365, 'learning_rate': 5.546206261394498e-05}

 49%|████▉     | 4920/10000 [33:15<16:43,  5.06it/s]
 49%|████▉     | 4921/10000 [33:16<17:05,  4.95it/s]
 49%|████▉     | 4922/10000 [33:16<17:05,  4.95it/s]
 49%|████▉     | 4923/10000 [33:16<16:55,  5.00it/s]
 49%|████▉     | 4924/10000 [33:16<16:46,  5.04it/s]
 49%|████▉     | 4925/10000 [33:16<16:44,  5.05it/s]
 49%|████▉     | 4926/10000 [33:17<16:23,  5.16it/s]
 49%|████▉     | 4927/10000 [33:17<16:20,  5.18it/s]
 49%|████▉     | 4928/10000 [33:17<16:16,  5.19it/s]
 49%|████▉     | 4929/10000 [33:17<16:18,  5.18it/s]
 49%|████▉     | 4930/10000 [33:17<16:22,  5.16it/s]
                                                    
{'loss': 0.1145, 'grad_norm': 0.8449073433876038, 'learning_rate': 5.529767562334437e-05}

 49%|████▉     | 4930/10000 [33:17<16:22,  5.16it/s]
 49%|████▉     | 4931/10000 [33:18<16:39,  5.07it/s]
 49%|████▉     | 4932/10000 [33:18<16:39,  5.07it/s]
 49%|████▉     | 4933/10000 [33:18<16:42,  5.05it/s]
 49%|████▉     | 4934/10000 [33:18<16:43,  5.05it/s]
 49%|████▉     | 4935/10000 [33:18<16:48,  5.02it/s]
 49%|████▉     | 4936/10000 [33:19<17:05,  4.94it/s]
 49%|████▉     | 4937/10000 [33:19<16:51,  5.01it/s]
 49%|████▉     | 4938/10000 [33:19<16:46,  5.03it/s]
 49%|████▉     | 4939/10000 [33:19<16:20,  5.16it/s]
 49%|████▉     | 4940/10000 [33:19<16:23,  5.14it/s]
                                                    
{'loss': 0.0948, 'grad_norm': 0.95359867811203, 'learning_rate': 5.5133230698211926e-05}

 49%|████▉     | 4940/10000 [33:19<16:23,  5.14it/s]
 49%|████▉     | 4941/10000 [33:20<16:47,  5.02it/s]
 49%|████▉     | 4942/10000 [33:20<17:20,  4.86it/s]
 49%|████▉     | 4943/10000 [33:20<17:39,  4.77it/s]
 49%|████▉     | 4944/10000 [33:20<17:37,  4.78it/s]
 49%|████▉     | 4945/10000 [33:20<17:28,  4.82it/s]
 49%|████▉     | 4946/10000 [33:21<17:15,  4.88it/s]
 49%|████▉     | 4947/10000 [33:21<16:51,  5.00it/s]
 49%|████▉     | 4948/10000 [33:21<16:23,  5.14it/s]
 49%|████▉     | 4949/10000 [33:21<16:02,  5.25it/s]
 50%|████▉     | 4950/10000 [33:21<15:47,  5.33it/s]
                                                    
{'loss': 0.1275, 'grad_norm': 0.9699293971061707, 'learning_rate': 5.496872963689096e-05}

 50%|████▉     | 4950/10000 [33:21<15:47,  5.33it/s]
 50%|████▉     | 4951/10000 [33:22<16:06,  5.23it/s]
 50%|████▉     | 4952/10000 [33:22<16:14,  5.18it/s]
 50%|████▉     | 4953/10000 [33:22<16:25,  5.12it/s]
 50%|████▉     | 4954/10000 [33:22<16:29,  5.10it/s]
 50%|████▉     | 4955/10000 [33:22<16:33,  5.08it/s]
 50%|████▉     | 4956/10000 [33:23<16:31,  5.09it/s]
 50%|████▉     | 4957/10000 [33:23<16:32,  5.08it/s]
 50%|████▉     | 4958/10000 [33:23<16:23,  5.13it/s]
 50%|████▉     | 4959/10000 [33:23<17:24,  4.83it/s]
 50%|████▉     | 4960/10000 [33:23<18:43,  4.48it/s]
                                                    
{'loss': 0.109, 'grad_norm': 0.7208645343780518, 'learning_rate': 5.4804174238338756e-05}

 50%|████▉     | 4960/10000 [33:23<18:43,  4.48it/s]
 50%|████▉     | 4961/10000 [33:24<19:34,  4.29it/s]
 50%|████▉     | 4962/10000 [33:24<19:20,  4.34it/s]
 50%|████▉     | 4963/10000 [33:24<20:02,  4.19it/s]
 50%|████▉     | 4964/10000 [33:24<18:56,  4.43it/s]
 50%|████▉     | 4965/10000 [33:25<17:51,  4.70it/s]
 50%|████▉     | 4966/10000 [33:25<17:22,  4.83it/s]
 50%|████▉     | 4967/10000 [33:25<17:04,  4.91it/s]
 50%|████▉     | 4968/10000 [33:25<17:36,  4.77it/s]
 50%|████▉     | 4969/10000 [33:25<18:27,  4.54it/s]
 50%|████▉     | 4970/10000 [33:26<18:24,  4.55it/s]
                                                    
{'loss': 0.0869, 'grad_norm': 0.8153486847877502, 'learning_rate': 5.463956630210678e-05}

 50%|████▉     | 4970/10000 [33:26<18:24,  4.55it/s]
 50%|████▉     | 4971/10000 [33:26<18:48,  4.46it/s]
 50%|████▉     | 4972/10000 [33:26<18:53,  4.44it/s]
 50%|████▉     | 4973/10000 [33:26<18:16,  4.58it/s]
 50%|████▉     | 4974/10000 [33:26<17:25,  4.81it/s]
 50%|████▉     | 4975/10000 [33:27<16:54,  4.95it/s]
 50%|████▉     | 4976/10000 [33:27<16:25,  5.10it/s]
 50%|████▉     | 4977/10000 [33:27<16:43,  5.00it/s]
 50%|████▉     | 4978/10000 [33:27<17:15,  4.85it/s]
 50%|████▉     | 4979/10000 [33:27<17:50,  4.69it/s]
 50%|████▉     | 4980/10000 [33:28<18:10,  4.60it/s]
                                                    
{'loss': 0.0892, 'grad_norm': 0.7621985673904419, 'learning_rate': 5.4474907628321046e-05}

 50%|████▉     | 4980/10000 [33:28<18:10,  4.60it/s]
 50%|████▉     | 4981/10000 [33:28<18:32,  4.51it/s]
 50%|████▉     | 4982/10000 [33:28<18:03,  4.63it/s]
 50%|████▉     | 4983/10000 [33:28<17:20,  4.82it/s]
 50%|████▉     | 4984/10000 [33:28<16:37,  5.03it/s]
 50%|████▉     | 4985/10000 [33:29<16:24,  5.09it/s]
 50%|████▉     | 4986/10000 [33:29<16:37,  5.02it/s]
 50%|████▉     | 4987/10000 [33:29<16:51,  4.96it/s]
 50%|████▉     | 4988/10000 [33:29<17:22,  4.81it/s]
 50%|████▉     | 4989/10000 [33:30<17:54,  4.66it/s]
 50%|████▉     | 4990/10000 [33:30<17:44,  4.71it/s]
                                                    
{'loss': 0.1055, 'grad_norm': 0.7622295022010803, 'learning_rate': 5.431020001766244e-05}

 50%|████▉     | 4990/10000 [33:30<17:44,  4.71it/s]
 50%|████▉     | 4991/10000 [33:30<18:04,  4.62it/s]
 50%|████▉     | 4992/10000 [33:30<17:33,  4.75it/s]
 50%|████▉     | 4993/10000 [33:30<16:59,  4.91it/s]
 50%|████▉     | 4994/10000 [33:31<16:46,  4.97it/s]
 50%|████▉     | 4995/10000 [33:31<16:55,  4.93it/s]
 50%|████▉     | 4996/10000 [33:31<17:19,  4.81it/s]
 50%|████▉     | 4997/10000 [33:31<17:38,  4.73it/s]
 50%|████▉     | 4998/10000 [33:31<17:58,  4.64it/s]
 50%|████▉     | 4999/10000 [33:32<18:12,  4.58it/s]
 50%|█████     | 5000/10000 [33:32<18:11,  4.58it/s]Rank 0, Worker 2: Caching shard...Rank 0, Worker 3: Caching shard...

Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.08 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.70 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.06 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.10 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.33 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.53 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 5000: eval/loss = 0.1138 (38s)

                                                    
{'loss': 0.0908, 'grad_norm': 0.7612760663032532, 'learning_rate': 5.4145445271346986e-05}

 50%|█████     | 5000/10000 [34:10<18:11,  4.58it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-5000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-5000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-5000/wandb_config.json

 50%|█████     | 5001/10000 [34:37<27:20:39, 19.69s/it]
 50%|█████     | 5002/10000 [34:37<19:12:35, 13.84s/it]
 50%|█████     | 5003/10000 [34:37<13:31:11,  9.74s/it]
 50%|█████     | 5004/10000 [34:38<9:32:24,  6.87s/it] 
 50%|█████     | 5005/10000 [34:38<6:45:22,  4.87s/it]
 50%|█████     | 5006/10000 [34:38<4:48:36,  3.47s/it]
 50%|█████     | 5007/10000 [34:38<3:26:46,  2.48s/it]
 50%|█████     | 5008/10000 [34:38<2:29:25,  1.80s/it]
 50%|█████     | 5009/10000 [34:39<1:49:15,  1.31s/it]
 50%|█████     | 5010/10000 [34:39<1:20:57,  1.03it/s]
                                                      
{'loss': 0.1023, 'grad_norm': 0.8658816814422607, 'learning_rate': 5.398064519110622e-05}

 50%|█████     | 5010/10000 [34:39<1:20:57,  1.03it/s]
 50%|█████     | 5011/10000 [34:39<1:01:13,  1.36it/s]
 50%|█████     | 5012/10000 [34:39<47:19,  1.76it/s]  
 50%|█████     | 5013/10000 [34:39<37:42,  2.20it/s]
 50%|█████     | 5014/10000 [34:39<31:34,  2.63it/s]
 50%|█████     | 5015/10000 [34:40<27:24,  3.03it/s]
 50%|█████     | 5016/10000 [34:40<24:27,  3.40it/s]
 50%|█████     | 5017/10000 [34:40<21:54,  3.79it/s]
 50%|█████     | 5018/10000 [34:40<20:04,  4.13it/s]
 50%|█████     | 5019/10000 [34:40<18:23,  4.51it/s]
 50%|█████     | 5020/10000 [34:41<17:12,  4.82it/s]
                                                    
{'loss': 0.1069, 'grad_norm': 0.874986469745636, 'learning_rate': 5.3815801579167394e-05}

 50%|█████     | 5020/10000 [34:41<17:12,  4.82it/s]
 50%|█████     | 5021/10000 [34:41<16:37,  4.99it/s]
 50%|█████     | 5022/10000 [34:41<16:34,  5.00it/s]
 50%|█████     | 5023/10000 [34:41<16:43,  4.96it/s]
 50%|█████     | 5024/10000 [34:41<17:22,  4.77it/s]
 50%|█████     | 5025/10000 [34:42<17:15,  4.80it/s]
 50%|█████     | 5026/10000 [34:42<17:00,  4.87it/s]
 50%|█████     | 5027/10000 [34:42<16:34,  5.00it/s]
 50%|█████     | 5028/10000 [34:42<16:03,  5.16it/s]
 50%|█████     | 5029/10000 [34:42<15:34,  5.32it/s]
 50%|█████     | 5030/10000 [34:43<15:33,  5.33it/s]
                                                    
{'loss': 0.0991, 'grad_norm': 0.9100071787834167, 'learning_rate': 5.365091623823382e-05}

 50%|█████     | 5030/10000 [34:43<15:33,  5.33it/s]
 50%|█████     | 5031/10000 [34:43<15:48,  5.24it/s]
 50%|█████     | 5032/10000 [34:43<16:48,  4.93it/s]
 50%|█████     | 5033/10000 [34:43<16:53,  4.90it/s]
 50%|█████     | 5034/10000 [34:43<16:52,  4.91it/s]
 50%|█████     | 5035/10000 [34:44<16:31,  5.01it/s]
 50%|█████     | 5036/10000 [34:44<16:14,  5.09it/s]
 50%|█████     | 5037/10000 [34:44<15:46,  5.24it/s]
 50%|█████     | 5038/10000 [34:44<15:42,  5.27it/s]
 50%|█████     | 5039/10000 [34:44<15:38,  5.29it/s]
 50%|█████     | 5040/10000 [34:44<15:33,  5.31it/s]
                                                    
{'loss': 0.0959, 'grad_norm': 0.8562759160995483, 'learning_rate': 5.348599097146521e-05}

 50%|█████     | 5040/10000 [34:45<15:33,  5.31it/s]
 50%|█████     | 5041/10000 [34:45<15:51,  5.21it/s]
 50%|█████     | 5042/10000 [34:45<15:39,  5.28it/s]
 50%|█████     | 5043/10000 [34:45<15:38,  5.28it/s]
 50%|█████     | 5044/10000 [34:45<15:45,  5.24it/s]
 50%|█████     | 5045/10000 [34:45<16:05,  5.13it/s]
 50%|█████     | 5046/10000 [34:46<16:27,  5.02it/s]
 50%|█████     | 5047/10000 [34:46<16:42,  4.94it/s]
 50%|█████     | 5048/10000 [34:46<16:45,  4.93it/s]
 50%|█████     | 5049/10000 [34:46<16:29,  5.01it/s]
 50%|█████     | 5050/10000 [34:46<16:04,  5.13it/s]
                                                    
{'loss': 0.1136, 'grad_norm': 0.8090814352035522, 'learning_rate': 5.3321027582457836e-05}

 50%|█████     | 5050/10000 [34:47<16:04,  5.13it/s]
 51%|█████     | 5051/10000 [34:47<16:16,  5.07it/s]
 51%|█████     | 5052/10000 [34:47<16:21,  5.04it/s]
 51%|█████     | 5053/10000 [34:47<16:26,  5.01it/s]
 51%|█████     | 5054/10000 [34:47<16:18,  5.05it/s]
 51%|█████     | 5055/10000 [34:47<16:18,  5.06it/s]
 51%|█████     | 5056/10000 [34:48<16:16,  5.06it/s]
 51%|█████     | 5057/10000 [34:48<15:55,  5.17it/s]
 51%|█████     | 5058/10000 [34:48<15:54,  5.18it/s]
 51%|█████     | 5059/10000 [34:48<16:37,  4.95it/s]
 51%|█████     | 5060/10000 [34:48<16:42,  4.93it/s]
                                                    
{'loss': 0.0845, 'grad_norm': 0.9330719113349915, 'learning_rate': 5.315602787522491e-05}

 51%|█████     | 5060/10000 [34:49<16:42,  4.93it/s]
 51%|█████     | 5061/10000 [34:49<17:38,  4.67it/s]
 51%|█████     | 5062/10000 [34:49<17:41,  4.65it/s]
 51%|█████     | 5063/10000 [34:49<17:40,  4.66it/s]
 51%|█████     | 5064/10000 [34:49<17:35,  4.68it/s]
 51%|█████     | 5065/10000 [34:50<17:06,  4.81it/s]
 51%|█████     | 5066/10000 [34:50<16:38,  4.94it/s]
 51%|█████     | 5067/10000 [34:50<16:16,  5.05it/s]
 51%|█████     | 5068/10000 [34:50<15:52,  5.18it/s]
 51%|█████     | 5069/10000 [34:50<16:31,  4.97it/s]
 51%|█████     | 5070/10000 [34:51<16:51,  4.87it/s]
                                                    
{'loss': 0.0904, 'grad_norm': 0.9269210696220398, 'learning_rate': 5.299099365417678e-05}

 51%|█████     | 5070/10000 [34:51<16:51,  4.87it/s]
 51%|█████     | 5071/10000 [34:51<17:15,  4.76it/s]
 51%|█████     | 5072/10000 [34:51<17:12,  4.77it/s]
 51%|█████     | 5073/10000 [34:51<17:31,  4.69it/s]
 51%|█████     | 5074/10000 [34:51<17:23,  4.72it/s]
 51%|█████     | 5075/10000 [34:52<17:03,  4.81it/s]
 51%|█████     | 5076/10000 [34:52<16:46,  4.89it/s]
 51%|█████     | 5077/10000 [34:52<16:33,  4.96it/s]
 51%|█████     | 5078/10000 [34:52<16:27,  4.98it/s]
 51%|█████     | 5079/10000 [34:52<16:29,  4.97it/s]
 51%|█████     | 5080/10000 [34:53<16:47,  4.88it/s]
                                                    
{'loss': 0.113, 'grad_norm': 0.9226139783859253, 'learning_rate': 5.2825926724101236e-05}

 51%|█████     | 5080/10000 [34:53<16:47,  4.88it/s]
 51%|█████     | 5081/10000 [34:53<17:01,  4.82it/s]
 51%|█████     | 5082/10000 [34:53<17:01,  4.81it/s]
 51%|█████     | 5083/10000 [34:53<17:06,  4.79it/s]
 51%|█████     | 5084/10000 [34:53<16:53,  4.85it/s]
 51%|█████     | 5085/10000 [34:54<16:27,  4.98it/s]
 51%|█████     | 5086/10000 [34:54<16:15,  5.04it/s]
 51%|█████     | 5087/10000 [34:54<15:58,  5.13it/s]
 51%|█████     | 5088/10000 [34:54<16:01,  5.11it/s]
 51%|█████     | 5089/10000 [34:54<16:11,  5.05it/s]
 51%|█████     | 5090/10000 [34:55<16:52,  4.85it/s]
                                                    
{'loss': 0.099, 'grad_norm': 0.7862170934677124, 'learning_rate': 5.26608288901438e-05}

 51%|█████     | 5090/10000 [34:55<16:52,  4.85it/s]
 51%|█████     | 5091/10000 [34:55<17:05,  4.79it/s]
 51%|█████     | 5092/10000 [34:55<16:56,  4.83it/s]
 51%|█████     | 5093/10000 [34:55<16:46,  4.88it/s]
 51%|█████     | 5094/10000 [34:55<16:34,  4.93it/s]
 51%|█████     | 5095/10000 [34:56<16:14,  5.03it/s]
 51%|█████     | 5096/10000 [34:56<16:05,  5.08it/s]
 51%|█████     | 5097/10000 [34:56<16:15,  5.02it/s]
 51%|█████     | 5098/10000 [34:56<16:27,  4.96it/s]
 51%|█████     | 5099/10000 [34:56<16:53,  4.84it/s]
 51%|█████     | 5100/10000 [34:57<17:51,  4.57it/s]
                                                    
{'loss': 0.1205, 'grad_norm': 0.7928630709648132, 'learning_rate': 5.24957019577879e-05}

 51%|█████     | 5100/10000 [34:57<17:51,  4.57it/s]
 51%|█████     | 5101/10000 [34:57<18:17,  4.46it/s]
 51%|█████     | 5102/10000 [34:57<18:11,  4.49it/s]
 51%|█████     | 5103/10000 [34:57<18:15,  4.47it/s]
 51%|█████     | 5104/10000 [34:58<18:08,  4.50it/s]
 51%|█████     | 5105/10000 [34:58<18:15,  4.47it/s]
 51%|█████     | 5106/10000 [34:58<18:53,  4.32it/s]
 51%|█████     | 5107/10000 [34:58<19:03,  4.28it/s]
 51%|█████     | 5108/10000 [34:59<19:03,  4.28it/s]
 51%|█████     | 5109/10000 [34:59<18:58,  4.30it/s]
 51%|█████     | 5110/10000 [34:59<18:28,  4.41it/s]
                                                    
{'loss': 0.0839, 'grad_norm': 0.9549708366394043, 'learning_rate': 5.2330547732835266e-05}

 51%|█████     | 5110/10000 [34:59<18:28,  4.41it/s]
 51%|█████     | 5111/10000 [34:59<21:28,  3.79it/s]
 51%|█████     | 5112/10000 [35:00<19:50,  4.11it/s]
 51%|█████     | 5113/10000 [35:00<18:52,  4.32it/s]
 51%|█████     | 5114/10000 [35:00<18:46,  4.34it/s]
 51%|█████     | 5115/10000 [35:00<18:11,  4.48it/s]
 51%|█████     | 5116/10000 [35:00<17:47,  4.57it/s]
 51%|█████     | 5117/10000 [35:01<17:13,  4.73it/s]
 51%|█████     | 5118/10000 [35:01<16:18,  4.99it/s]
 51%|█████     | 5119/10000 [35:01<15:43,  5.17it/s]
 51%|█████     | 5120/10000 [35:01<15:49,  5.14it/s]
                                                    
{'loss': 0.0946, 'grad_norm': 1.108726978302002, 'learning_rate': 5.2165368021385996e-05}

 51%|█████     | 5120/10000 [35:01<15:49,  5.14it/s]
 51%|█████     | 5121/10000 [35:01<16:39,  4.88it/s]
 51%|█████     | 5122/10000 [35:02<16:55,  4.80it/s]
 51%|█████     | 5123/10000 [35:02<17:13,  4.72it/s]
 51%|█████     | 5124/10000 [35:02<17:22,  4.68it/s]
 51%|█████▏    | 5125/10000 [35:02<17:06,  4.75it/s]
 51%|█████▏    | 5126/10000 [35:02<16:49,  4.83it/s]
 51%|█████▏    | 5127/10000 [35:03<16:40,  4.87it/s]
 51%|█████▏    | 5128/10000 [35:03<16:57,  4.79it/s]
 51%|█████▏    | 5129/10000 [35:03<17:13,  4.71it/s]
 51%|█████▏    | 5130/10000 [35:03<18:03,  4.49it/s]
                                                    
{'loss': 0.0781, 'grad_norm': 0.7452027201652527, 'learning_rate': 5.200016462981897e-05}

 51%|█████▏    | 5130/10000 [35:03<18:03,  4.49it/s]
 51%|█████▏    | 5131/10000 [35:04<17:58,  4.51it/s]
 51%|█████▏    | 5132/10000 [35:04<17:28,  4.64it/s]
 51%|█████▏    | 5133/10000 [35:04<17:19,  4.68it/s]
 51%|█████▏    | 5134/10000 [35:04<17:35,  4.61it/s]
 51%|█████▏    | 5135/10000 [35:04<17:02,  4.76it/s]
 51%|█████▏    | 5136/10000 [35:05<17:03,  4.75it/s]
 51%|█████▏    | 5137/10000 [35:05<17:07,  4.73it/s]
 51%|█████▏    | 5138/10000 [35:05<17:55,  4.52it/s]
 51%|█████▏    | 5139/10000 [35:05<18:02,  4.49it/s]
 51%|█████▏    | 5140/10000 [35:05<18:01,  4.49it/s]
                                                    
{'loss': 0.1071, 'grad_norm': 0.9842017292976379, 'learning_rate': 5.1834939364772015e-05}

 51%|█████▏    | 5140/10000 [35:06<18:01,  4.49it/s]
 51%|█████▏    | 5141/10000 [35:06<18:40,  4.34it/s]
 51%|█████▏    | 5142/10000 [35:06<19:17,  4.20it/s]
 51%|█████▏    | 5143/10000 [35:06<19:15,  4.20it/s]
 51%|█████▏    | 5144/10000 [35:06<18:53,  4.28it/s]
 51%|█████▏    | 5145/10000 [35:07<18:03,  4.48it/s]
 51%|█████▏    | 5146/10000 [35:07<17:44,  4.56it/s]
 51%|█████▏    | 5147/10000 [35:07<17:44,  4.56it/s]
 51%|█████▏    | 5148/10000 [35:07<17:55,  4.51it/s]
 51%|█████▏    | 5149/10000 [35:08<18:06,  4.46it/s]
 52%|█████▏    | 5150/10000 [35:08<17:48,  4.54it/s]
                                                    
{'loss': 0.1238, 'grad_norm': 0.7672536373138428, 'learning_rate': 5.166969403312214e-05}

 52%|█████▏    | 5150/10000 [35:08<17:48,  4.54it/s]
 52%|█████▏    | 5151/10000 [35:08<18:21,  4.40it/s]
 52%|█████▏    | 5152/10000 [35:08<18:13,  4.43it/s]
 52%|█████▏    | 5153/10000 [35:08<17:42,  4.56it/s]
 52%|█████▏    | 5154/10000 [35:09<17:20,  4.66it/s]
 52%|█████▏    | 5155/10000 [35:09<16:49,  4.80it/s]
 52%|█████▏    | 5156/10000 [35:09<16:36,  4.86it/s]
 52%|█████▏    | 5157/10000 [35:09<17:10,  4.70it/s]
 52%|█████▏    | 5158/10000 [35:09<17:25,  4.63it/s]
 52%|█████▏    | 5159/10000 [35:10<17:41,  4.56it/s]
 52%|█████▏    | 5160/10000 [35:10<17:41,  4.56it/s]
                                                    
{'loss': 0.135, 'grad_norm': 0.6951093077659607, 'learning_rate': 5.1504430441965844e-05}

 52%|█████▏    | 5160/10000 [35:10<17:41,  4.56it/s]
 52%|█████▏    | 5161/10000 [35:10<17:39,  4.57it/s]
 52%|█████▏    | 5162/10000 [35:10<17:19,  4.65it/s]
 52%|█████▏    | 5163/10000 [35:11<16:49,  4.79it/s]
 52%|█████▏    | 5164/10000 [35:11<16:20,  4.93it/s]
 52%|█████▏    | 5165/10000 [35:11<16:15,  4.96it/s]
 52%|█████▏    | 5166/10000 [35:11<16:56,  4.75it/s]
 52%|█████▏    | 5167/10000 [35:11<17:07,  4.71it/s]
 52%|█████▏    | 5168/10000 [35:12<17:29,  4.60it/s]
 52%|█████▏    | 5169/10000 [35:12<17:32,  4.59it/s]
 52%|█████▏    | 5170/10000 [35:12<17:36,  4.57it/s]
                                                    
{'loss': 0.1017, 'grad_norm': 0.912887454032898, 'learning_rate': 5.133915039859923e-05}

 52%|█████▏    | 5170/10000 [35:12<17:36,  4.57it/s]
 52%|█████▏    | 5171/10000 [35:12<17:37,  4.57it/s]
 52%|█████▏    | 5172/10000 [35:12<17:11,  4.68it/s]
 52%|█████▏    | 5173/10000 [35:13<16:58,  4.74it/s]Rank 0, Worker 1: Wait for shard 71 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 52%|█████▏    | 5174/10000 [35:13<17:10,  4.69it/s]
 52%|█████▏    | 5175/10000 [35:13<17:33,  4.58it/s]
 52%|█████▏    | 5176/10000 [35:13<18:03,  4.45it/s]
 52%|█████▏    | 5177/10000 [35:14<18:31,  4.34it/s]
 52%|█████▏    | 5178/10000 [35:14<18:27,  4.35it/s]
 52%|█████▏    | 5179/10000 [35:14<18:57,  4.24it/s]
 52%|█████▏    | 5180/10000 [35:14<18:56,  4.24it/s]
                                                    
{'loss': 0.0804, 'grad_norm': 0.9999566674232483, 'learning_rate': 5.1173855710498444e-05}

 52%|█████▏    | 5180/10000 [35:14<18:56,  4.24it/s]
 52%|█████▏    | 5181/10000 [35:15<18:53,  4.25it/s]
 52%|█████▏    | 5182/10000 [35:15<18:57,  4.24it/s]
 52%|█████▏    | 5183/10000 [35:15<19:12,  4.18it/s]
 52%|█████▏    | 5184/10000 [35:15<19:16,  4.17it/s]
 52%|█████▏    | 5185/10000 [35:15<18:39,  4.30it/s]
 52%|█████▏    | 5186/10000 [35:16<18:23,  4.36it/s]
 52%|█████▏    | 5187/10000 [35:16<18:04,  4.44it/s]
 52%|█████▏    | 5188/10000 [35:16<17:38,  4.55it/s]
 52%|█████▏    | 5189/10000 [35:16<17:24,  4.61it/s]
 52%|█████▏    | 5190/10000 [35:17<17:15,  4.64it/s]
                                                    
{'loss': 0.1112, 'grad_norm': 0.7690122127532959, 'learning_rate': 5.100854818529967e-05}

 52%|█████▏    | 5190/10000 [35:17<17:15,  4.64it/s]
 52%|█████▏    | 5191/10000 [35:17<17:37,  4.55it/s]
 52%|█████▏    | 5192/10000 [35:17<17:45,  4.51it/s]
 52%|█████▏    | 5193/10000 [35:17<18:08,  4.42it/s]
 52%|█████▏    | 5194/10000 [35:17<18:03,  4.44it/s]
 52%|█████▏    | 5195/10000 [35:18<18:29,  4.33it/s]
 52%|█████▏    | 5196/10000 [35:18<18:31,  4.32it/s]
 52%|█████▏    | 5197/10000 [35:18<18:05,  4.42it/s]
 52%|█████▏    | 5198/10000 [35:18<17:32,  4.56it/s]
 52%|█████▏    | 5199/10000 [35:19<17:16,  4.63it/s]
 52%|█████▏    | 5200/10000 [35:19<17:09,  4.66it/s]
                                                    
{'loss': 0.0909, 'grad_norm': 0.7402052879333496, 'learning_rate': 5.084322963077951e-05}

 52%|█████▏    | 5200/10000 [35:19<17:09,  4.66it/s]
 52%|█████▏    | 5201/10000 [35:19<17:09,  4.66it/s]
 52%|█████▏    | 5202/10000 [35:19<17:09,  4.66it/s]
 52%|█████▏    | 5203/10000 [35:19<16:58,  4.71it/s]
 52%|█████▏    | 5204/10000 [35:20<16:58,  4.71it/s]
 52%|█████▏    | 5205/10000 [35:20<17:03,  4.69it/s]
 52%|█████▏    | 5206/10000 [35:20<17:14,  4.64it/s]
 52%|█████▏    | 5207/10000 [35:20<16:56,  4.72it/s]
 52%|█████▏    | 5208/10000 [35:20<16:43,  4.78it/s]
 52%|█████▏    | 5209/10000 [35:21<16:54,  4.72it/s]
 52%|█████▏    | 5210/10000 [35:21<16:45,  4.76it/s]
                                                    
{'loss': 0.1168, 'grad_norm': 0.9506291151046753, 'learning_rate': 5.067790185483522e-05}

 52%|█████▏    | 5210/10000 [35:21<16:45,  4.76it/s]
 52%|█████▏    | 5211/10000 [35:21<16:30,  4.84it/s]
 52%|█████▏    | 5212/10000 [35:21<16:09,  4.94it/s]
 52%|█████▏    | 5213/10000 [35:21<15:34,  5.12it/s]
 52%|█████▏    | 5214/10000 [35:22<15:48,  5.05it/s]
 52%|█████▏    | 5215/10000 [35:22<15:41,  5.08it/s]
 52%|█████▏    | 5216/10000 [35:22<15:37,  5.10it/s]
 52%|█████▏    | 5217/10000 [35:22<15:36,  5.11it/s]
 52%|█████▏    | 5218/10000 [35:22<15:28,  5.15it/s]
 52%|█████▏    | 5219/10000 [35:23<15:03,  5.29it/s]
 52%|█████▏    | 5220/10000 [35:23<14:56,  5.33it/s]
                                                    
{'loss': 0.0758, 'grad_norm': 0.6709886789321899, 'learning_rate': 5.0512566665464844e-05}

 52%|█████▏    | 5220/10000 [35:23<14:56,  5.33it/s]
 52%|█████▏    | 5221/10000 [35:23<14:59,  5.32it/s]
 52%|█████▏    | 5222/10000 [35:23<14:52,  5.35it/s]
 52%|█████▏    | 5223/10000 [35:23<14:56,  5.33it/s]
 52%|█████▏    | 5224/10000 [35:24<14:39,  5.43it/s]
 52%|█████▏    | 5225/10000 [35:24<14:47,  5.38it/s]
 52%|█████▏    | 5226/10000 [35:24<14:50,  5.36it/s]
 52%|█████▏    | 5227/10000 [35:24<14:47,  5.38it/s]
 52%|█████▏    | 5228/10000 [35:24<14:46,  5.39it/s]
 52%|█████▏    | 5229/10000 [35:24<14:44,  5.39it/s]
 52%|█████▏    | 5230/10000 [35:25<14:33,  5.46it/s]
                                                    
{'loss': 0.0987, 'grad_norm': 0.8839654326438904, 'learning_rate': 5.034722587074755e-05}

 52%|█████▏    | 5230/10000 [35:25<14:33,  5.46it/s]
 52%|█████▏    | 5231/10000 [35:25<14:39,  5.42it/s]
 52%|█████▏    | 5232/10000 [35:25<14:31,  5.47it/s]
 52%|█████▏    | 5233/10000 [35:25<14:31,  5.47it/s]
 52%|█████▏    | 5234/10000 [35:25<14:23,  5.52it/s]
 52%|█████▏    | 5235/10000 [35:26<14:43,  5.39it/s]
 52%|█████▏    | 5236/10000 [35:26<14:44,  5.39it/s]
 52%|█████▏    | 5237/10000 [35:26<14:50,  5.35it/s]
 52%|█████▏    | 5238/10000 [35:26<14:55,  5.32it/s]
 52%|█████▏    | 5239/10000 [35:26<14:53,  5.33it/s]
 52%|█████▏    | 5240/10000 [35:27<14:47,  5.36it/s]
                                                    
{'loss': 0.1018, 'grad_norm': 0.7217310070991516, 'learning_rate': 5.018188127882375e-05}

 52%|█████▏    | 5240/10000 [35:27<14:47,  5.36it/s]
 52%|█████▏    | 5241/10000 [35:27<14:48,  5.35it/s]
 52%|█████▏    | 5242/10000 [35:27<14:28,  5.48it/s]
 52%|█████▏    | 5243/10000 [35:27<14:19,  5.54it/s]
 52%|█████▏    | 5244/10000 [35:27<14:13,  5.57it/s]
 52%|█████▏    | 5245/10000 [35:27<14:06,  5.62it/s]
 52%|█████▏    | 5246/10000 [35:28<14:13,  5.57it/s]
 52%|█████▏    | 5247/10000 [35:28<14:27,  5.48it/s]
 52%|█████▏    | 5248/10000 [35:28<14:26,  5.48it/s]
 52%|█████▏    | 5249/10000 [35:28<14:16,  5.55it/s]
 52%|█████▎    | 5250/10000 [35:28<14:09,  5.59it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...Rank 0, Worker 2: Caching shard...

Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.39 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.72 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 16.73 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.14 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.16 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.24 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 5250: eval/loss = 0.1016 (34s)

                                                    
{'loss': 0.0923, 'grad_norm': 0.6325317025184631, 'learning_rate': 5.0016534697875417e-05}

 52%|█████▎    | 5250/10000 [36:02<14:09,  5.59it/s]
 53%|█████▎    | 5251/10000 [36:03<13:43:14, 10.40s/it]
 53%|█████▎    | 5252/10000 [36:03<9:40:21,  7.33s/it] 
 53%|█████▎    | 5253/10000 [36:03<6:50:19,  5.19s/it]
 53%|█████▎    | 5254/10000 [36:03<4:51:34,  3.69s/it]
 53%|█████▎    | 5255/10000 [36:03<3:28:28,  2.64s/it]
 53%|█████▎    | 5256/10000 [36:03<2:30:18,  1.90s/it]
 53%|█████▎    | 5257/10000 [36:04<1:50:11,  1.39s/it]
 53%|█████▎    | 5258/10000 [36:04<1:22:15,  1.04s/it]
 53%|█████▎    | 5259/10000 [36:04<1:02:28,  1.26it/s]
 53%|█████▎    | 5260/10000 [36:04<48:59,  1.61it/s]  
                                                    
{'loss': 0.0936, 'grad_norm': 0.9298528432846069, 'learning_rate': 4.9851187936106294e-05}

 53%|█████▎    | 5260/10000 [36:04<48:59,  1.61it/s]
 53%|█████▎    | 5261/10000 [36:05<38:46,  2.04it/s]
 53%|█████▎    | 5262/10000 [36:05<31:18,  2.52it/s]
 53%|█████▎    | 5263/10000 [36:05<26:09,  3.02it/s]
 53%|█████▎    | 5264/10000 [36:05<22:27,  3.51it/s]
 53%|█████▎    | 5265/10000 [36:05<20:11,  3.91it/s]
 53%|█████▎    | 5266/10000 [36:05<18:44,  4.21it/s]
 53%|█████▎    | 5267/10000 [36:06<17:57,  4.39it/s]
 53%|█████▎    | 5268/10000 [36:06<17:15,  4.57it/s]
 53%|█████▎    | 5269/10000 [36:06<16:38,  4.74it/s]
 53%|█████▎    | 5270/10000 [36:06<15:57,  4.94it/s]
                                                    
{'loss': 0.1009, 'grad_norm': 0.8486387729644775, 'learning_rate': 4.968584280172206e-05}

 53%|█████▎    | 5270/10000 [36:06<15:57,  4.94it/s]
 53%|█████▎    | 5271/10000 [36:06<15:39,  5.03it/s]
 53%|█████▎    | 5272/10000 [36:07<15:10,  5.20it/s]
 53%|█████▎    | 5273/10000 [36:07<15:00,  5.25it/s]
 53%|█████▎    | 5274/10000 [36:07<14:57,  5.26it/s]
 53%|█████▎    | 5275/10000 [36:07<15:08,  5.20it/s]
 53%|█████▎    | 5276/10000 [36:07<15:07,  5.20it/s]
 53%|█████▎    | 5277/10000 [36:08<14:55,  5.27it/s]
 53%|█████▎    | 5278/10000 [36:08<14:48,  5.31it/s]
 53%|█████▎    | 5279/10000 [36:08<14:34,  5.40it/s]Rank 0, Worker 5: Wait for shard 67 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 53%|█████▎    | 5280/10000 [36:08<14:48,  5.31it/s]
                                                    
{'loss': 0.0823, 'grad_norm': 0.719869077205658, 'learning_rate': 4.95205011029106e-05}

 53%|█████▎    | 5280/10000 [36:08<14:48,  5.31it/s]
 53%|█████▎    | 5281/10000 [36:08<15:09,  5.19it/s]
 53%|█████▎    | 5282/10000 [36:08<15:22,  5.11it/s]
 53%|█████▎    | 5283/10000 [36:09<15:26,  5.09it/s]
 53%|█████▎    | 5284/10000 [36:09<15:33,  5.05it/s]
 53%|█████▎    | 5285/10000 [36:09<15:35,  5.04it/s]
 53%|█████▎    | 5286/10000 [36:09<15:38,  5.02it/s]
 53%|█████▎    | 5287/10000 [36:09<15:28,  5.08it/s]
 53%|█████▎    | 5288/10000 [36:10<15:31,  5.06it/s]
 53%|█████▎    | 5289/10000 [36:10<15:44,  4.99it/s]
 53%|█████▎    | 5290/10000 [36:10<16:08,  4.86it/s]
                                                    
{'loss': 0.0958, 'grad_norm': 0.7929495573043823, 'learning_rate': 4.935516464782227e-05}

 53%|█████▎    | 5290/10000 [36:10<16:08,  4.86it/s]
 53%|█████▎    | 5291/10000 [36:10<16:48,  4.67it/s]
 53%|█████▎    | 5292/10000 [36:11<16:58,  4.62it/s]
 53%|█████▎    | 5293/10000 [36:11<16:57,  4.63it/s]
 53%|█████▎    | 5294/10000 [36:11<16:39,  4.71it/s]
 53%|█████▎    | 5295/10000 [36:11<16:37,  4.72it/s]
 53%|█████▎    | 5296/10000 [36:11<16:35,  4.72it/s]Rank 0, Worker 4: Wait for shard 78 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 53%|█████▎    | 5297/10000 [36:12<16:56,  4.63it/s]
 53%|█████▎    | 5298/10000 [36:12<16:53,  4.64it/s]
 53%|█████▎    | 5299/10000 [36:12<17:09,  4.57it/s]
 53%|█████▎    | 5300/10000 [36:12<17:21,  4.51it/s]
                                                    
{'loss': 0.0928, 'grad_norm': 0.9161084294319153, 'learning_rate': 4.918983524455003e-05}

 53%|█████▎    | 5300/10000 [36:12<17:21,  4.51it/s]
 53%|█████▎    | 5301/10000 [36:12<17:03,  4.59it/s]
 53%|█████▎    | 5302/10000 [36:13<16:52,  4.64it/s]
 53%|█████▎    | 5303/10000 [36:13<17:05,  4.58it/s]
 53%|█████▎    | 5304/10000 [36:13<17:36,  4.44it/s]
 53%|█████▎    | 5305/10000 [36:13<17:54,  4.37it/s]
 53%|█████▎    | 5306/10000 [36:14<18:36,  4.21it/s]
 53%|█████▎    | 5307/10000 [36:14<18:58,  4.12it/s]
 53%|█████▎    | 5308/10000 [36:14<17:55,  4.36it/s]
 53%|█████▎    | 5309/10000 [36:14<17:09,  4.56it/s]
 53%|█████▎    | 5310/10000 [36:15<16:48,  4.65it/s]
                                                    
{'loss': 0.0993, 'grad_norm': 0.783526599407196, 'learning_rate': 4.9024514701109766e-05}

 53%|█████▎    | 5310/10000 [36:15<16:48,  4.65it/s]
 53%|█████▎    | 5311/10000 [36:15<16:47,  4.66it/s]
 53%|█████▎    | 5312/10000 [36:15<17:21,  4.50it/s]
 53%|█████▎    | 5313/10000 [36:15<17:42,  4.41it/s]
 53%|█████▎    | 5314/10000 [36:15<18:21,  4.26it/s]
 53%|█████▎    | 5315/10000 [36:16<18:02,  4.33it/s]
 53%|█████▎    | 5316/10000 [36:16<17:51,  4.37it/s]
 53%|█████▎    | 5317/10000 [36:16<17:17,  4.51it/s]
 53%|█████▎    | 5318/10000 [36:16<16:47,  4.64it/s]
 53%|█████▎    | 5319/10000 [36:16<16:00,  4.87it/s]Rank 0, Worker 3: Wait for shard 60 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 53%|█████▎    | 5320/10000 [36:17<15:57,  4.89it/s]
                                                    
{'loss': 0.0927, 'grad_norm': 0.8160496950149536, 'learning_rate': 4.885920482542043e-05}

 53%|█████▎    | 5320/10000 [36:17<15:57,  4.89it/s]
 53%|█████▎    | 5321/10000 [36:17<16:17,  4.79it/s]
 53%|█████▎    | 5322/10000 [36:17<16:37,  4.69it/s]
 53%|█████▎    | 5323/10000 [36:17<17:34,  4.43it/s]
 53%|█████▎    | 5324/10000 [36:18<18:50,  4.14it/s]
 53%|█████▎    | 5325/10000 [36:18<18:11,  4.28it/s]
 53%|█████▎    | 5326/10000 [36:18<18:05,  4.31it/s]
 53%|█████▎    | 5327/10000 [36:18<17:24,  4.47it/s]
 53%|█████▎    | 5328/10000 [36:19<16:43,  4.66it/s]
 53%|█████▎    | 5329/10000 [36:19<16:24,  4.74it/s]
 53%|█████▎    | 5330/10000 [36:19<16:12,  4.80it/s]
                                                    
{'loss': 0.0947, 'grad_norm': 1.007938265800476, 'learning_rate': 4.869390742528438e-05}

 53%|█████▎    | 5330/10000 [36:19<16:12,  4.80it/s]
 53%|█████▎    | 5331/10000 [36:19<16:21,  4.76it/s]
 53%|█████▎    | 5332/10000 [36:19<16:32,  4.70it/s]
 53%|█████▎    | 5333/10000 [36:20<16:46,  4.63it/s]
 53%|█████▎    | 5334/10000 [36:20<17:24,  4.47it/s]
 53%|█████▎    | 5335/10000 [36:20<17:55,  4.34it/s]
 53%|█████▎    | 5336/10000 [36:20<17:21,  4.48it/s]
 53%|█████▎    | 5337/10000 [36:20<16:24,  4.73it/s]
 53%|█████▎    | 5338/10000 [36:21<16:00,  4.85it/s]
 53%|█████▎    | 5339/10000 [36:21<15:44,  4.93it/s]
 53%|█████▎    | 5340/10000 [36:21<16:10,  4.80it/s]
                                                    
{'loss': 0.1022, 'grad_norm': 0.7827715873718262, 'learning_rate': 4.852862430836744e-05}

 53%|█████▎    | 5340/10000 [36:21<16:10,  4.80it/s]
 53%|█████▎    | 5341/10000 [36:21<17:23,  4.46it/s]
 53%|█████▎    | 5342/10000 [36:22<17:49,  4.36it/s]
 53%|█████▎    | 5343/10000 [36:22<17:49,  4.36it/s]
 53%|█████▎    | 5344/10000 [36:22<17:13,  4.50it/s]
 53%|█████▎    | 5345/10000 [36:22<16:20,  4.75it/s]
 53%|█████▎    | 5346/10000 [36:22<15:57,  4.86it/s]
 53%|█████▎    | 5347/10000 [36:23<16:11,  4.79it/s]
 53%|█████▎    | 5348/10000 [36:23<16:49,  4.61it/s]
 53%|█████▎    | 5349/10000 [36:23<17:07,  4.53it/s]
 54%|█████▎    | 5350/10000 [36:23<17:42,  4.38it/s]
                                                    
{'loss': 0.1021, 'grad_norm': 0.7112279534339905, 'learning_rate': 4.836335728217933e-05}

 54%|█████▎    | 5350/10000 [36:23<17:42,  4.38it/s]
 54%|█████▎    | 5351/10000 [36:24<18:13,  4.25it/s]
 54%|█████▎    | 5352/10000 [36:24<17:44,  4.37it/s]
 54%|█████▎    | 5353/10000 [36:24<16:39,  4.65it/s]
 54%|█████▎    | 5354/10000 [36:24<15:52,  4.88it/s]
 54%|█████▎    | 5355/10000 [36:24<15:34,  4.97it/s]
 54%|█████▎    | 5356/10000 [36:25<15:49,  4.89it/s]
 54%|█████▎    | 5357/10000 [36:25<16:23,  4.72it/s]
 54%|█████▎    | 5358/10000 [36:25<17:01,  4.54it/s]
 54%|█████▎    | 5359/10000 [36:25<17:55,  4.31it/s]
 54%|█████▎    | 5360/10000 [36:25<17:32,  4.41it/s]
                                                    
{'loss': 0.076, 'grad_norm': 0.7599149346351624, 'learning_rate': 4.819810815405379e-05}

 54%|█████▎    | 5360/10000 [36:26<17:32,  4.41it/s]
 54%|█████▎    | 5361/10000 [36:26<16:46,  4.61it/s]
 54%|█████▎    | 5362/10000 [36:26<15:54,  4.86it/s]
 54%|█████▎    | 5363/10000 [36:26<15:25,  5.01it/s]
 54%|█████▎    | 5364/10000 [36:26<16:52,  4.58it/s]
 54%|█████▎    | 5365/10000 [36:27<16:30,  4.68it/s]
 54%|█████▎    | 5366/10000 [36:27<16:28,  4.69it/s]
 54%|█████▎    | 5367/10000 [36:27<16:27,  4.69it/s]
 54%|█████▎    | 5368/10000 [36:27<15:58,  4.83it/s]
 54%|█████▎    | 5369/10000 [36:27<15:12,  5.08it/s]
 54%|█████▎    | 5370/10000 [36:27<14:42,  5.25it/s]
                                                    
{'loss': 0.0894, 'grad_norm': 0.9205085039138794, 'learning_rate': 4.803287873112877e-05}

 54%|█████▎    | 5370/10000 [36:28<14:42,  5.25it/s]
 54%|█████▎    | 5371/10000 [36:28<14:57,  5.16it/s]
 54%|█████▎    | 5372/10000 [36:28<15:11,  5.08it/s]Rank 0, Worker 2: Wait for shard 26 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 54%|█████▎    | 5373/10000 [36:28<15:44,  4.90it/s]
 54%|█████▎    | 5374/10000 [36:28<16:08,  4.78it/s]
 54%|█████▍    | 5375/10000 [36:29<15:54,  4.85it/s]
 54%|█████▍    | 5376/10000 [36:29<15:47,  4.88it/s]
 54%|█████▍    | 5377/10000 [36:29<15:26,  4.99it/s]
 54%|█████▍    | 5378/10000 [36:29<14:55,  5.16it/s]
 54%|█████▍    | 5379/10000 [36:29<14:36,  5.27it/s]
 54%|█████▍    | 5380/10000 [36:29<14:58,  5.14it/s]
                                                    
{'loss': 0.0947, 'grad_norm': 0.716441810131073, 'learning_rate': 4.786767082032681e-05}

 54%|█████▍    | 5380/10000 [36:30<14:58,  5.14it/s]
 54%|█████▍    | 5381/10000 [36:30<15:39,  4.92it/s]
 54%|█████▍    | 5382/10000 [36:30<16:11,  4.75it/s]
 54%|█████▍    | 5383/10000 [36:30<16:21,  4.71it/s]
 54%|█████▍    | 5384/10000 [36:30<16:43,  4.60it/s]
 54%|█████▍    | 5385/10000 [36:31<16:56,  4.54it/s]
 54%|█████▍    | 5386/10000 [36:31<16:31,  4.65it/s]
 54%|█████▍    | 5387/10000 [36:31<16:33,  4.64it/s]
 54%|█████▍    | 5388/10000 [36:31<16:07,  4.77it/s]
 54%|█████▍    | 5389/10000 [36:31<15:32,  4.95it/s]
 54%|█████▍    | 5390/10000 [36:32<15:25,  4.98it/s]
                                                    
{'loss': 0.079, 'grad_norm': 0.7823354005813599, 'learning_rate': 4.77024862283351e-05}

 54%|█████▍    | 5390/10000 [36:32<15:25,  4.98it/s]
 54%|█████▍    | 5391/10000 [36:32<15:59,  4.80it/s]
 54%|█████▍    | 5392/10000 [36:32<15:56,  4.82it/s]
 54%|█████▍    | 5393/10000 [36:32<15:58,  4.81it/s]
 54%|█████▍    | 5394/10000 [36:32<16:01,  4.79it/s]
 54%|█████▍    | 5395/10000 [36:33<16:01,  4.79it/s]
 54%|█████▍    | 5396/10000 [36:33<16:16,  4.71it/s]
 54%|█████▍    | 5397/10000 [36:33<16:36,  4.62it/s]
 54%|█████▍    | 5398/10000 [36:33<16:01,  4.79it/s]
 54%|█████▍    | 5399/10000 [36:33<15:23,  4.98it/s]
 54%|█████▍    | 5400/10000 [36:34<15:13,  5.04it/s]
                                                    
{'loss': 0.0915, 'grad_norm': 0.8588849306106567, 'learning_rate': 4.753732676158593e-05}

 54%|█████▍    | 5400/10000 [36:34<15:13,  5.04it/s]
 54%|█████▍    | 5401/10000 [36:34<15:26,  4.96it/s]
 54%|█████▍    | 5402/10000 [36:34<15:23,  4.98it/s]
 54%|█████▍    | 5403/10000 [36:34<15:29,  4.95it/s]
 54%|█████▍    | 5404/10000 [36:34<15:35,  4.91it/s]
 54%|█████▍    | 5405/10000 [36:35<15:34,  4.92it/s]
 54%|█████▍    | 5406/10000 [36:35<15:25,  4.96it/s]
 54%|█████▍    | 5407/10000 [36:35<15:14,  5.02it/s]
 54%|█████▍    | 5408/10000 [36:35<15:05,  5.07it/s]
 54%|█████▍    | 5409/10000 [36:35<14:55,  5.13it/s]
 54%|█████▍    | 5410/10000 [36:36<15:06,  5.06it/s]
                                                    
{'loss': 0.0859, 'grad_norm': 1.0715975761413574, 'learning_rate': 4.737219422623672e-05}

 54%|█████▍    | 5410/10000 [36:36<15:06,  5.06it/s]
 54%|█████▍    | 5411/10000 [36:36<15:13,  5.02it/s]
 54%|█████▍    | 5412/10000 [36:36<15:10,  5.04it/s]
 54%|█████▍    | 5413/10000 [36:36<15:34,  4.91it/s]
 54%|█████▍    | 5414/10000 [36:36<15:41,  4.87it/s]
 54%|█████▍    | 5415/10000 [36:37<15:42,  4.86it/s]
 54%|█████▍    | 5416/10000 [36:37<15:43,  4.86it/s]
 54%|█████▍    | 5417/10000 [36:37<15:31,  4.92it/s]
 54%|█████▍    | 5418/10000 [36:37<15:18,  4.99it/s]
 54%|█████▍    | 5419/10000 [36:37<15:14,  5.01it/s]
 54%|█████▍    | 5420/10000 [36:38<15:14,  5.01it/s]
                                                    
{'loss': 0.094, 'grad_norm': 0.8519423007965088, 'learning_rate': 4.720709042815044e-05}

 54%|█████▍    | 5420/10000 [36:38<15:14,  5.01it/s]
 54%|█████▍    | 5421/10000 [36:38<15:47,  4.83it/s]
 54%|█████▍    | 5422/10000 [36:38<15:50,  4.82it/s]
 54%|█████▍    | 5423/10000 [36:38<15:41,  4.86it/s]
 54%|█████▍    | 5424/10000 [36:39<15:40,  4.87it/s]
 54%|█████▍    | 5425/10000 [36:39<15:38,  4.87it/s]
 54%|█████▍    | 5426/10000 [36:39<15:36,  4.89it/s]
 54%|█████▍    | 5427/10000 [36:39<15:24,  4.95it/s]
 54%|█████▍    | 5428/10000 [36:39<15:30,  4.91it/s]
 54%|█████▍    | 5429/10000 [36:40<15:37,  4.88it/s]
 54%|█████▍    | 5430/10000 [36:40<15:37,  4.87it/s]
                                                    
{'loss': 0.0812, 'grad_norm': 0.8101336359977722, 'learning_rate': 4.704201717287578e-05}

 54%|█████▍    | 5430/10000 [36:40<15:37,  4.87it/s]
 54%|█████▍    | 5431/10000 [36:40<16:02,  4.75it/s]
 54%|█████▍    | 5432/10000 [36:40<15:56,  4.78it/s]
 54%|█████▍    | 5433/10000 [36:40<15:44,  4.84it/s]
 54%|█████▍    | 5434/10000 [36:41<15:08,  5.03it/s]
 54%|█████▍    | 5435/10000 [36:41<14:44,  5.16it/s]
 54%|█████▍    | 5436/10000 [36:41<14:45,  5.15it/s]
 54%|█████▍    | 5437/10000 [36:41<14:54,  5.10it/s]
 54%|█████▍    | 5438/10000 [36:41<15:10,  5.01it/s]
 54%|█████▍    | 5439/10000 [36:42<15:25,  4.93it/s]
 54%|█████▍    | 5440/10000 [36:42<15:35,  4.87it/s]
                                                    
{'loss': 0.1178, 'grad_norm': 0.7211766242980957, 'learning_rate': 4.6876976265627404e-05}

 54%|█████▍    | 5440/10000 [36:42<15:35,  4.87it/s]
 54%|█████▍    | 5441/10000 [36:42<15:48,  4.81it/s]
 54%|█████▍    | 5442/10000 [36:42<15:23,  4.94it/s]
 54%|█████▍    | 5443/10000 [36:42<14:56,  5.08it/s]
 54%|█████▍    | 5444/10000 [36:43<14:49,  5.12it/s]
 54%|█████▍    | 5445/10000 [36:43<14:53,  5.10it/s]
 54%|█████▍    | 5446/10000 [36:43<15:29,  4.90it/s]
 54%|█████▍    | 5447/10000 [36:43<15:55,  4.76it/s]
 54%|█████▍    | 5448/10000 [36:43<16:44,  4.53it/s]Rank 0, Worker 0: Wait for shard 72 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 54%|█████▍    | 5449/10000 [36:44<17:20,  4.38it/s]
 55%|█████▍    | 5450/10000 [36:44<17:29,  4.33it/s]
                                                    
{'loss': 0.0834, 'grad_norm': 0.7692839503288269, 'learning_rate': 4.671196951126626e-05}

 55%|█████▍    | 5450/10000 [36:44<17:29,  4.33it/s]
 55%|█████▍    | 5451/10000 [36:44<17:53,  4.24it/s]
 55%|█████▍    | 5452/10000 [36:44<17:53,  4.24it/s]
 55%|█████▍    | 5453/10000 [36:45<16:56,  4.47it/s]
 55%|█████▍    | 5454/10000 [36:45<16:26,  4.61it/s]
 55%|█████▍    | 5455/10000 [36:45<16:07,  4.70it/s]
 55%|█████▍    | 5456/10000 [36:45<15:50,  4.78it/s]
 55%|█████▍    | 5457/10000 [36:45<16:15,  4.66it/s]
 55%|█████▍    | 5458/10000 [36:46<16:13,  4.66it/s]
 55%|█████▍    | 5459/10000 [36:46<16:32,  4.58it/s]
 55%|█████▍    | 5460/10000 [36:46<16:26,  4.60it/s]
                                                    
{'loss': 0.0818, 'grad_norm': 0.7380297780036926, 'learning_rate': 4.654699871427971e-05}

 55%|█████▍    | 5460/10000 [36:46<16:26,  4.60it/s]
 55%|█████▍    | 5461/10000 [36:46<16:30,  4.58it/s]
 55%|█████▍    | 5462/10000 [36:47<16:26,  4.60it/s]
 55%|█████▍    | 5463/10000 [36:47<16:07,  4.69it/s]
 55%|█████▍    | 5464/10000 [36:47<15:54,  4.75it/s]
 55%|█████▍    | 5465/10000 [36:47<15:40,  4.82it/s]
 55%|█████▍    | 5466/10000 [36:47<15:47,  4.79it/s]
 55%|█████▍    | 5467/10000 [36:48<15:48,  4.78it/s]
 55%|█████▍    | 5468/10000 [36:48<15:52,  4.76it/s]
 55%|█████▍    | 5469/10000 [36:48<16:05,  4.69it/s]
 55%|█████▍    | 5470/10000 [36:48<16:19,  4.62it/s]
                                                    
{'loss': 0.0839, 'grad_norm': 0.7921284437179565, 'learning_rate': 4.6382065678762034e-05}

 55%|█████▍    | 5470/10000 [36:48<16:19,  4.62it/s]
 55%|█████▍    | 5471/10000 [36:48<17:09,  4.40it/s]
 55%|█████▍    | 5472/10000 [36:49<17:24,  4.34it/s]
 55%|█████▍    | 5473/10000 [36:49<16:54,  4.46it/s]
 55%|█████▍    | 5474/10000 [36:49<16:27,  4.58it/s]
 55%|█████▍    | 5475/10000 [36:49<16:11,  4.66it/s]
 55%|█████▍    | 5476/10000 [36:50<16:08,  4.67it/s]
 55%|█████▍    | 5477/10000 [36:50<16:26,  4.59it/s]
 55%|█████▍    | 5478/10000 [36:50<16:53,  4.46it/s]
 55%|█████▍    | 5479/10000 [36:50<17:31,  4.30it/s]
 55%|█████▍    | 5480/10000 [36:51<17:56,  4.20it/s]
                                                    
{'loss': 0.0827, 'grad_norm': 0.7934443950653076, 'learning_rate': 4.6217172208394424e-05}

 55%|█████▍    | 5480/10000 [36:51<17:56,  4.20it/s]
 55%|█████▍    | 5481/10000 [36:51<18:00,  4.18it/s]
 55%|█████▍    | 5482/10000 [36:51<17:29,  4.30it/s]
 55%|█████▍    | 5483/10000 [36:51<17:39,  4.26it/s]
 55%|█████▍    | 5484/10000 [36:51<17:26,  4.31it/s]
 55%|█████▍    | 5485/10000 [36:52<16:47,  4.48it/s]
 55%|█████▍    | 5486/10000 [36:52<17:15,  4.36it/s]
 55%|█████▍    | 5487/10000 [36:52<17:43,  4.24it/s]
 55%|█████▍    | 5488/10000 [36:52<18:05,  4.16it/s]
 55%|█████▍    | 5489/10000 [36:53<18:10,  4.14it/s]
 55%|█████▍    | 5490/10000 [36:53<17:20,  4.34it/s]
                                                    
{'loss': 0.1038, 'grad_norm': 1.083687663078308, 'learning_rate': 4.605232010642549e-05}

 55%|█████▍    | 5490/10000 [36:53<17:20,  4.34it/s]
 55%|█████▍    | 5491/10000 [36:53<16:50,  4.46it/s]
 55%|█████▍    | 5492/10000 [36:53<16:25,  4.57it/s]
 55%|█████▍    | 5493/10000 [36:53<16:19,  4.60it/s]
 55%|█████▍    | 5494/10000 [36:54<16:10,  4.64it/s]
 55%|█████▍    | 5495/10000 [36:54<16:21,  4.59it/s]
 55%|█████▍    | 5496/10000 [36:54<16:53,  4.44it/s]
 55%|█████▍    | 5497/10000 [36:54<17:14,  4.35it/s]
 55%|█████▍    | 5498/10000 [36:55<16:56,  4.43it/s]
 55%|█████▍    | 5499/10000 [36:55<16:16,  4.61it/s]
 55%|█████▌    | 5500/10000 [36:55<15:45,  4.76it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Caching shard...Rank 0, Worker 0: Caching shard...

Rank 0, Worker 2: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.63 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.89 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 18.94 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.27 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.27 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.42 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 5500: eval/loss = 0.1001 (37s)

                                                    
{'loss': 0.0767, 'grad_norm': 0.9212234616279602, 'learning_rate': 4.588751117565142e-05}

 55%|█████▌    | 5500/10000 [37:32<15:45,  4.76it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-5500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-5500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-5500/wandb_config.json

 55%|█████▌    | 5501/10000 [38:00<24:29:24, 19.60s/it]
 55%|█████▌    | 5502/10000 [38:00<17:12:45, 13.78s/it]
 55%|█████▌    | 5503/10000 [38:00<12:07:25,  9.71s/it]
 55%|█████▌    | 5504/10000 [38:01<8:40:23,  6.94s/it] 
 55%|█████▌    | 5505/10000 [38:01<6:08:21,  4.92s/it]
 55%|█████▌    | 5506/10000 [38:01<4:21:59,  3.50s/it]
 55%|█████▌    | 5507/10000 [38:01<3:07:40,  2.51s/it]
 55%|█████▌    | 5508/10000 [38:01<2:15:37,  1.81s/it]
 55%|█████▌    | 5509/10000 [38:02<1:39:31,  1.33s/it]
 55%|█████▌    | 5510/10000 [38:02<1:14:12,  1.01it/s]
                                                      
{'loss': 0.0886, 'grad_norm': 0.9997414946556091, 'learning_rate': 4.5722747218396214e-05}

 55%|█████▌    | 5510/10000 [38:02<1:14:12,  1.01it/s]
 55%|█████▌    | 5511/10000 [38:02<56:45,  1.32it/s]  
 55%|█████▌    | 5512/10000 [38:02<44:14,  1.69it/s]
 55%|█████▌    | 5513/10000 [38:02<35:04,  2.13it/s]
 55%|█████▌    | 5514/10000 [38:03<28:32,  2.62it/s]
 55%|█████▌    | 5515/10000 [38:03<24:10,  3.09it/s]
 55%|█████▌    | 5516/10000 [38:03<21:05,  3.54it/s]
 55%|█████▌    | 5517/10000 [38:03<19:23,  3.85it/s]
 55%|█████▌    | 5518/10000 [38:03<18:03,  4.14it/s]
 55%|█████▌    | 5519/10000 [38:04<17:11,  4.35it/s]
 55%|█████▌    | 5520/10000 [38:04<16:32,  4.51it/s]
                                                    
{'loss': 0.0821, 'grad_norm': 0.7122896909713745, 'learning_rate': 4.5558030036492194e-05}

 55%|█████▌    | 5520/10000 [38:04<16:32,  4.51it/s]
 55%|█████▌    | 5521/10000 [38:04<16:17,  4.58it/s]
 55%|█████▌    | 5522/10000 [38:04<15:40,  4.76it/s]
 55%|█████▌    | 5523/10000 [38:04<14:59,  4.98it/s]
 55%|█████▌    | 5524/10000 [38:05<14:33,  5.12it/s]
 55%|█████▌    | 5525/10000 [38:05<14:16,  5.23it/s]
 55%|█████▌    | 5526/10000 [38:05<14:00,  5.32it/s]
 55%|█████▌    | 5527/10000 [38:05<13:53,  5.37it/s]
 55%|█████▌    | 5528/10000 [38:05<14:02,  5.31it/s]
 55%|█████▌    | 5529/10000 [38:06<14:14,  5.24it/s]
 55%|█████▌    | 5530/10000 [38:06<14:48,  5.03it/s]
                                                    
{'loss': 0.0775, 'grad_norm': 0.7129685878753662, 'learning_rate': 4.539336143125999e-05}

 55%|█████▌    | 5530/10000 [38:06<14:48,  5.03it/s]
 55%|█████▌    | 5531/10000 [38:06<14:48,  5.03it/s]
 55%|█████▌    | 5532/10000 [38:06<14:21,  5.18it/s]
 55%|█████▌    | 5533/10000 [38:06<14:03,  5.29it/s]
 55%|█████▌    | 5534/10000 [38:07<13:57,  5.33it/s]
 55%|█████▌    | 5535/10000 [38:07<13:40,  5.44it/s]
 55%|█████▌    | 5536/10000 [38:07<13:43,  5.42it/s]
 55%|█████▌    | 5537/10000 [38:07<13:51,  5.37it/s]
 55%|█████▌    | 5538/10000 [38:07<13:56,  5.34it/s]
 55%|█████▌    | 5539/10000 [38:07<13:46,  5.40it/s]
 55%|█████▌    | 5540/10000 [38:08<13:37,  5.46it/s]
                                                    
{'loss': 0.0763, 'grad_norm': 0.6717953085899353, 'learning_rate': 4.522874320348916e-05}

 55%|█████▌    | 5540/10000 [38:08<13:37,  5.46it/s]
 55%|█████▌    | 5541/10000 [38:08<13:39,  5.44it/s]
 55%|█████▌    | 5542/10000 [38:08<13:39,  5.44it/s]
 55%|█████▌    | 5543/10000 [38:08<13:27,  5.52it/s]
 55%|█████▌    | 5544/10000 [38:08<13:23,  5.54it/s]
 55%|█████▌    | 5545/10000 [38:09<13:43,  5.41it/s]
 55%|█████▌    | 5546/10000 [38:09<14:27,  5.13it/s]
 55%|█████▌    | 5547/10000 [38:09<14:59,  4.95it/s]
 55%|█████▌    | 5548/10000 [38:09<15:30,  4.78it/s]
 55%|█████▌    | 5549/10000 [38:09<15:42,  4.72it/s]
 56%|█████▌    | 5550/10000 [38:10<15:53,  4.67it/s]
                                                    
{'loss': 0.1166, 'grad_norm': 0.7830241918563843, 'learning_rate': 4.506417715341821e-05}

 56%|█████▌    | 5550/10000 [38:10<15:53,  4.67it/s]
 56%|█████▌    | 5551/10000 [38:10<15:41,  4.73it/s]
 56%|█████▌    | 5552/10000 [38:10<15:03,  4.92it/s]
 56%|█████▌    | 5553/10000 [38:10<14:53,  4.97it/s]
 56%|█████▌    | 5554/10000 [38:10<14:49,  5.00it/s]
 56%|█████▌    | 5555/10000 [38:11<14:25,  5.14it/s]
 56%|█████▌    | 5556/10000 [38:11<14:49,  4.99it/s]
 56%|█████▌    | 5557/10000 [38:11<15:12,  4.87it/s]
 56%|█████▌    | 5558/10000 [38:11<15:21,  4.82it/s]
 56%|█████▌    | 5559/10000 [38:11<15:30,  4.77it/s]
 56%|█████▌    | 5560/10000 [38:12<15:38,  4.73it/s]
                                                    
{'loss': 0.0728, 'grad_norm': 0.78812575340271, 'learning_rate': 4.489966508071511e-05}

 56%|█████▌    | 5560/10000 [38:12<15:38,  4.73it/s]
 56%|█████▌    | 5561/10000 [38:12<15:48,  4.68it/s]
 56%|█████▌    | 5562/10000 [38:12<15:38,  4.73it/s]
 56%|█████▌    | 5563/10000 [38:12<15:10,  4.87it/s]
 56%|█████▌    | 5564/10000 [38:12<14:40,  5.04it/s]
 56%|█████▌    | 5565/10000 [38:13<14:25,  5.12it/s]
 56%|█████▌    | 5566/10000 [38:13<14:35,  5.06it/s]
 56%|█████▌    | 5567/10000 [38:13<14:59,  4.93it/s]
 56%|█████▌    | 5568/10000 [38:13<15:24,  4.79it/s]
 56%|█████▌    | 5569/10000 [38:14<15:48,  4.67it/s]
 56%|█████▌    | 5570/10000 [38:14<15:50,  4.66it/s]
                                                    
{'loss': 0.08, 'grad_norm': 0.7623831033706665, 'learning_rate': 4.4735208784457575e-05}

 56%|█████▌    | 5570/10000 [38:14<15:50,  4.66it/s]
 56%|█████▌    | 5571/10000 [38:14<15:43,  4.69it/s]
 56%|█████▌    | 5572/10000 [38:14<15:02,  4.90it/s]
 56%|█████▌    | 5573/10000 [38:14<14:37,  5.05it/s]
 56%|█████▌    | 5574/10000 [38:14<14:20,  5.14it/s]
 56%|█████▌    | 5575/10000 [38:15<14:27,  5.10it/s]
 56%|█████▌    | 5576/10000 [38:15<14:53,  4.95it/s]
 56%|█████▌    | 5577/10000 [38:15<15:34,  4.73it/s]
 56%|█████▌    | 5578/10000 [38:15<16:01,  4.60it/s]
 56%|█████▌    | 5579/10000 [38:16<15:54,  4.63it/s]
 56%|█████▌    | 5580/10000 [38:16<15:45,  4.67it/s]
                                                    
{'loss': 0.1045, 'grad_norm': 0.6921680569648743, 'learning_rate': 4.457081006311325e-05}

 56%|█████▌    | 5580/10000 [38:16<15:45,  4.67it/s]
 56%|█████▌    | 5581/10000 [38:16<15:19,  4.81it/s]
 56%|█████▌    | 5582/10000 [38:16<14:41,  5.01it/s]
 56%|█████▌    | 5583/10000 [38:16<14:34,  5.05it/s]
 56%|█████▌    | 5584/10000 [38:17<14:50,  4.96it/s]
 56%|█████▌    | 5585/10000 [38:17<15:35,  4.72it/s]
 56%|█████▌    | 5586/10000 [38:17<15:33,  4.73it/s]
 56%|█████▌    | 5587/10000 [38:17<15:58,  4.61it/s]
 56%|█████▌    | 5588/10000 [38:18<16:40,  4.41it/s]
 56%|█████▌    | 5589/10000 [38:18<15:52,  4.63it/s]
 56%|█████▌    | 5590/10000 [38:18<15:28,  4.75it/s]
                                                    
{'loss': 0.0841, 'grad_norm': 0.8056199550628662, 'learning_rate': 4.440647071452027e-05}

 56%|█████▌    | 5590/10000 [38:18<15:28,  4.75it/s]
 56%|█████▌    | 5591/10000 [38:18<15:52,  4.63it/s]
 56%|█████▌    | 5592/10000 [38:18<15:42,  4.68it/s]
 56%|█████▌    | 5593/10000 [38:19<15:40,  4.69it/s]
 56%|█████▌    | 5594/10000 [38:19<16:40,  4.41it/s]
 56%|█████▌    | 5595/10000 [38:19<16:18,  4.50it/s]
 56%|█████▌    | 5596/10000 [38:19<16:25,  4.47it/s]
 56%|█████▌    | 5597/10000 [38:19<16:24,  4.47it/s]
 56%|█████▌    | 5598/10000 [38:20<16:25,  4.46it/s]
 56%|█████▌    | 5599/10000 [38:20<16:58,  4.32it/s]
 56%|█████▌    | 5600/10000 [38:20<17:40,  4.15it/s]
                                                    
{'loss': 0.0898, 'grad_norm': 0.7235461473464966, 'learning_rate': 4.424219253586737e-05}

 56%|█████▌    | 5600/10000 [38:20<17:40,  4.15it/s]
 56%|█████▌    | 5601/10000 [38:20<18:04,  4.06it/s]
 56%|█████▌    | 5602/10000 [38:21<18:14,  4.02it/s]
 56%|█████▌    | 5603/10000 [38:21<18:16,  4.01it/s]
 56%|█████▌    | 5604/10000 [38:21<17:30,  4.19it/s]
 56%|█████▌    | 5605/10000 [38:21<16:51,  4.34it/s]
 56%|█████▌    | 5606/10000 [38:22<16:47,  4.36it/s]
 56%|█████▌    | 5607/10000 [38:22<17:29,  4.19it/s]
 56%|█████▌    | 5608/10000 [38:22<16:55,  4.33it/s]
 56%|█████▌    | 5609/10000 [38:22<16:59,  4.31it/s]
 56%|█████▌    | 5610/10000 [38:23<16:45,  4.37it/s]
                                                    
{'loss': 0.1022, 'grad_norm': 0.8183784484863281, 'learning_rate': 4.407797732367443e-05}

 56%|█████▌    | 5610/10000 [38:23<16:45,  4.37it/s]
 56%|█████▌    | 5611/10000 [38:23<16:53,  4.33it/s]
 56%|█████▌    | 5612/10000 [38:23<16:09,  4.53it/s]
 56%|█████▌    | 5613/10000 [38:23<15:32,  4.70it/s]
 56%|█████▌    | 5614/10000 [38:23<15:14,  4.79it/s]
 56%|█████▌    | 5615/10000 [38:24<15:22,  4.75it/s]
 56%|█████▌    | 5616/10000 [38:24<15:25,  4.74it/s]
 56%|█████▌    | 5617/10000 [38:24<15:59,  4.57it/s]
 56%|█████▌    | 5618/10000 [38:24<16:27,  4.44it/s]
 56%|█████▌    | 5619/10000 [38:25<16:37,  4.39it/s]
 56%|█████▌    | 5620/10000 [38:25<16:26,  4.44it/s]
                                                    
{'loss': 0.0996, 'grad_norm': 0.7427429556846619, 'learning_rate': 4.391382687377268e-05}

 56%|█████▌    | 5620/10000 [38:25<16:26,  4.44it/s]
 56%|█████▌    | 5621/10000 [38:25<16:41,  4.37it/s]
 56%|█████▌    | 5622/10000 [38:25<16:22,  4.46it/s]
 56%|█████▌    | 5623/10000 [38:25<16:00,  4.56it/s]
 56%|█████▌    | 5624/10000 [38:26<15:58,  4.57it/s]
 56%|█████▋    | 5625/10000 [38:26<16:16,  4.48it/s]
 56%|█████▋    | 5626/10000 [38:26<16:36,  4.39it/s]
 56%|█████▋    | 5627/10000 [38:26<16:52,  4.32it/s]
 56%|█████▋    | 5628/10000 [38:27<17:06,  4.26it/s]
 56%|█████▋    | 5629/10000 [38:27<16:56,  4.30it/s]
 56%|█████▋    | 5630/10000 [38:27<16:58,  4.29it/s]
                                                    
{'loss': 0.0877, 'grad_norm': 0.6695629954338074, 'learning_rate': 4.374974298128512e-05}

 56%|█████▋    | 5630/10000 [38:27<16:58,  4.29it/s]
 56%|█████▋    | 5631/10000 [38:27<16:47,  4.33it/s]
 56%|█████▋    | 5632/10000 [38:27<16:26,  4.43it/s]
 56%|█████▋    | 5633/10000 [38:28<16:15,  4.48it/s]
 56%|█████▋    | 5634/10000 [38:28<16:05,  4.52it/s]
 56%|█████▋    | 5635/10000 [38:28<15:51,  4.59it/s]
 56%|█████▋    | 5636/10000 [38:28<15:44,  4.62it/s]
 56%|█████▋    | 5637/10000 [38:29<15:38,  4.65it/s]
 56%|█████▋    | 5638/10000 [38:29<15:27,  4.70it/s]
 56%|█████▋    | 5639/10000 [38:29<15:30,  4.69it/s]
 56%|█████▋    | 5640/10000 [38:29<15:19,  4.74it/s]
                                                    
{'loss': 0.1232, 'grad_norm': 0.9349445104598999, 'learning_rate': 4.358572744060699e-05}

 56%|█████▋    | 5640/10000 [38:29<15:19,  4.74it/s]
 56%|█████▋    | 5641/10000 [38:29<15:28,  4.69it/s]
 56%|█████▋    | 5642/10000 [38:30<15:23,  4.72it/s]
 56%|█████▋    | 5643/10000 [38:30<15:13,  4.77it/s]
 56%|█████▋    | 5644/10000 [38:30<15:20,  4.73it/s]
 56%|█████▋    | 5645/10000 [38:30<15:38,  4.64it/s]
 56%|█████▋    | 5646/10000 [38:30<15:33,  4.66it/s]
 56%|█████▋    | 5647/10000 [38:31<15:29,  4.68it/s]
 56%|█████▋    | 5648/10000 [38:31<15:28,  4.69it/s]
 56%|█████▋    | 5649/10000 [38:31<15:13,  4.76it/s]
 56%|█████▋    | 5650/10000 [38:31<15:29,  4.68it/s]
                                                    
{'loss': 0.0869, 'grad_norm': 0.8588927984237671, 'learning_rate': 4.342178204538588e-05}

 56%|█████▋    | 5650/10000 [38:31<15:29,  4.68it/s]
 57%|█████▋    | 5651/10000 [38:32<15:36,  4.64it/s]
 57%|█████▋    | 5652/10000 [38:32<15:35,  4.65it/s]
 57%|█████▋    | 5653/10000 [38:32<15:49,  4.58it/s]
 57%|█████▋    | 5654/10000 [38:32<15:43,  4.61it/s]
 57%|█████▋    | 5655/10000 [38:32<15:22,  4.71it/s]
 57%|█████▋    | 5656/10000 [38:33<15:10,  4.77it/s]
 57%|█████▋    | 5657/10000 [38:33<15:03,  4.81it/s]
 57%|█████▋    | 5658/10000 [38:33<15:18,  4.73it/s]
 57%|█████▋    | 5659/10000 [38:33<15:28,  4.68it/s]
 57%|█████▋    | 5660/10000 [38:33<15:18,  4.73it/s]
                                                    
{'loss': 0.0799, 'grad_norm': 0.786688506603241, 'learning_rate': 4.325790858850241e-05}

 57%|█████▋    | 5660/10000 [38:33<15:18,  4.73it/s]
 57%|█████▋    | 5661/10000 [38:34<15:30,  4.66it/s]
 57%|█████▋    | 5662/10000 [38:34<15:21,  4.71it/s]
 57%|█████▋    | 5663/10000 [38:34<15:16,  4.73it/s]
 57%|█████▋    | 5664/10000 [38:34<15:31,  4.65it/s]
 57%|█████▋    | 5665/10000 [38:34<15:35,  4.63it/s]
 57%|█████▋    | 5666/10000 [38:35<15:31,  4.65it/s]
 57%|█████▋    | 5667/10000 [38:35<15:24,  4.69it/s]
 57%|█████▋    | 5668/10000 [38:35<15:08,  4.77it/s]
 57%|█████▋    | 5669/10000 [38:35<14:56,  4.83it/s]
 57%|█████▋    | 5670/10000 [38:36<14:45,  4.89it/s]
                                                    
{'loss': 0.0828, 'grad_norm': 0.6458063721656799, 'learning_rate': 4.309410886205043e-05}

 57%|█████▋    | 5670/10000 [38:36<14:45,  4.89it/s]
 57%|█████▋    | 5671/10000 [38:36<14:54,  4.84it/s]
 57%|█████▋    | 5672/10000 [38:36<14:53,  4.84it/s]
 57%|█████▋    | 5673/10000 [38:36<14:53,  4.84it/s]
 57%|█████▋    | 5674/10000 [38:36<14:56,  4.83it/s]
 57%|█████▋    | 5675/10000 [38:37<14:51,  4.85it/s]
 57%|█████▋    | 5676/10000 [38:37<14:41,  4.90it/s]
 57%|█████▋    | 5677/10000 [38:37<14:31,  4.96it/s]
 57%|█████▋    | 5678/10000 [38:37<14:27,  4.98it/s]
 57%|█████▋    | 5679/10000 [38:37<14:09,  5.09it/s]
 57%|█████▋    | 5680/10000 [38:38<14:15,  5.05it/s]
                                                    
{'loss': 0.1017, 'grad_norm': 0.9166744947433472, 'learning_rate': 4.293038465731752e-05}

 57%|█████▋    | 5680/10000 [38:38<14:15,  5.05it/s]
 57%|█████▋    | 5681/10000 [38:38<14:33,  4.94it/s]
 57%|█████▋    | 5682/10000 [38:38<14:41,  4.90it/s]
 57%|█████▋    | 5683/10000 [38:38<14:56,  4.82it/s]
 57%|█████▋    | 5684/10000 [38:38<15:14,  4.72it/s]
 57%|█████▋    | 5685/10000 [38:39<15:10,  4.74it/s]
 57%|█████▋    | 5686/10000 [38:39<15:32,  4.63it/s]
 57%|█████▋    | 5687/10000 [38:39<15:10,  4.74it/s]
 57%|█████▋    | 5688/10000 [38:39<15:07,  4.75it/s]
 57%|█████▋    | 5689/10000 [38:39<15:27,  4.65it/s]
 57%|█████▋    | 5690/10000 [38:40<15:14,  4.71it/s]
                                                    
{'loss': 0.084, 'grad_norm': 0.9064713716506958, 'learning_rate': 4.276673776476533e-05}

 57%|█████▋    | 5690/10000 [38:40<15:14,  4.71it/s]
 57%|█████▋    | 5691/10000 [38:40<15:14,  4.71it/s]
 57%|█████▋    | 5692/10000 [38:40<15:00,  4.78it/s]
 57%|█████▋    | 5693/10000 [38:40<14:40,  4.89it/s]
 57%|█████▋    | 5694/10000 [38:40<14:12,  5.05it/s]
 57%|█████▋    | 5695/10000 [38:41<13:48,  5.20it/s]
 57%|█████▋    | 5696/10000 [38:41<13:53,  5.17it/s]
 57%|█████▋    | 5697/10000 [38:41<13:49,  5.19it/s]
 57%|█████▋    | 5698/10000 [38:41<13:55,  5.15it/s]
 57%|█████▋    | 5699/10000 [38:41<13:56,  5.14it/s]
 57%|█████▋    | 5700/10000 [38:42<14:09,  5.06it/s]
                                                    
{'loss': 0.1037, 'grad_norm': 0.7649261355400085, 'learning_rate': 4.260316997401007e-05}

 57%|█████▋    | 5700/10000 [38:42<14:09,  5.06it/s]
 57%|█████▋    | 5701/10000 [38:42<14:27,  4.96it/s]
 57%|█████▋    | 5702/10000 [38:42<13:51,  5.17it/s]
 57%|█████▋    | 5703/10000 [38:42<13:37,  5.25it/s]
 57%|█████▋    | 5704/10000 [38:42<13:40,  5.24it/s]
 57%|█████▋    | 5705/10000 [38:43<13:33,  5.28it/s]
 57%|█████▋    | 5706/10000 [38:43<13:25,  5.33it/s]
 57%|█████▋    | 5707/10000 [38:43<13:29,  5.30it/s]
 57%|█████▋    | 5708/10000 [38:43<13:09,  5.44it/s]
 57%|█████▋    | 5709/10000 [38:43<13:01,  5.49it/s]
 57%|█████▋    | 5710/10000 [38:43<12:59,  5.50it/s]
                                                    
{'loss': 0.0806, 'grad_norm': 0.7940148115158081, 'learning_rate': 4.243968307380293e-05}

 57%|█████▋    | 5710/10000 [38:44<12:59,  5.50it/s]
 57%|█████▋    | 5711/10000 [38:44<13:25,  5.32it/s]
 57%|█████▋    | 5712/10000 [38:44<13:24,  5.33it/s]
 57%|█████▋    | 5713/10000 [38:44<13:25,  5.32it/s]
 57%|█████▋    | 5714/10000 [38:44<13:18,  5.37it/s]
 57%|█████▋    | 5715/10000 [38:44<13:18,  5.37it/s]
 57%|█████▋    | 5716/10000 [38:45<13:15,  5.38it/s]
 57%|█████▋    | 5717/10000 [38:45<13:01,  5.48it/s]
 57%|█████▋    | 5718/10000 [38:45<12:52,  5.54it/s]
 57%|█████▋    | 5719/10000 [38:45<12:46,  5.58it/s]
 57%|█████▋    | 5720/10000 [38:45<12:39,  5.63it/s]
                                                    
{'loss': 0.0876, 'grad_norm': 0.7147015929222107, 'learning_rate': 4.22762788520104e-05}

 57%|█████▋    | 5720/10000 [38:45<12:39,  5.63it/s]
 57%|█████▋    | 5721/10000 [38:46<13:03,  5.46it/s]
 57%|█████▋    | 5722/10000 [38:46<13:10,  5.41it/s]
 57%|█████▋    | 5723/10000 [38:46<13:18,  5.36it/s]
 57%|█████▋    | 5724/10000 [38:46<13:26,  5.30it/s]
 57%|█████▋    | 5725/10000 [38:46<13:23,  5.32it/s]
 57%|█████▋    | 5726/10000 [38:46<13:16,  5.36it/s]
 57%|█████▋    | 5727/10000 [38:47<13:00,  5.47it/s]
 57%|█████▋    | 5728/10000 [38:47<12:52,  5.53it/s]
 57%|█████▋    | 5729/10000 [38:47<12:44,  5.58it/s]
 57%|█████▋    | 5730/10000 [38:47<12:40,  5.61it/s]
                                                    
{'loss': 0.104, 'grad_norm': 0.6901019811630249, 'learning_rate': 4.211295909559491e-05}

 57%|█████▋    | 5730/10000 [38:47<12:40,  5.61it/s]
 57%|█████▋    | 5731/10000 [38:47<12:57,  5.49it/s]
 57%|█████▋    | 5732/10000 [38:48<13:20,  5.33it/s]
 57%|█████▋    | 5733/10000 [38:48<13:46,  5.17it/s]
 57%|█████▋    | 5734/10000 [38:48<14:02,  5.07it/s]
 57%|█████▋    | 5735/10000 [38:48<13:47,  5.15it/s]
 57%|█████▋    | 5736/10000 [38:48<13:33,  5.24it/s]
 57%|█████▋    | 5737/10000 [38:49<13:19,  5.34it/s]
 57%|█████▋    | 5738/10000 [38:49<13:05,  5.43it/s]
 57%|█████▋    | 5739/10000 [38:49<13:02,  5.44it/s]
 57%|█████▋    | 5740/10000 [38:49<13:10,  5.39it/s]
                                                    
{'loss': 0.0863, 'grad_norm': 0.8219081163406372, 'learning_rate': 4.194972559059511e-05}

 57%|█████▋    | 5740/10000 [38:49<13:10,  5.39it/s]
 57%|█████▋    | 5741/10000 [38:49<13:27,  5.27it/s]
 57%|█████▋    | 5742/10000 [38:49<13:35,  5.22it/s]
 57%|█████▋    | 5743/10000 [38:50<13:50,  5.12it/s]
 57%|█████▋    | 5744/10000 [38:50<13:48,  5.13it/s]
 57%|█████▋    | 5745/10000 [38:50<13:42,  5.17it/s]
 57%|█████▋    | 5746/10000 [38:50<14:26,  4.91it/s]
 57%|█████▋    | 5747/10000 [38:50<13:39,  5.19it/s]
 57%|█████▋    | 5748/10000 [38:51<13:21,  5.31it/s]
 57%|█████▋    | 5749/10000 [38:51<13:04,  5.42it/s]
 57%|█████▊    | 5750/10000 [38:51<12:51,  5.51it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 4: Caching shard...
Rank 0, Worker 3: Caching shard...

Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.38 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 16.69 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.72 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 16.89 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.02 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.27 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 5750: eval/loss = 0.1024 (35s)

                                                    
{'loss': 0.0854, 'grad_norm': 0.7481513023376465, 'learning_rate': 4.178658012210651e-05}

 57%|█████▊    | 5750/10000 [39:26<12:51,  5.51it/s]
 58%|█████▊    | 5751/10000 [39:26<12:33:29, 10.64s/it]
 58%|█████▊    | 5752/10000 [39:26<8:51:30,  7.51s/it] 
 58%|█████▊    | 5753/10000 [39:26<6:16:04,  5.31s/it]
 58%|█████▊    | 5754/10000 [39:27<4:27:19,  3.78s/it]
 58%|█████▊    | 5755/10000 [39:27<3:11:03,  2.70s/it]
 58%|█████▊    | 5756/10000 [39:27<2:17:29,  1.94s/it]
 58%|█████▊    | 5757/10000 [39:27<1:39:54,  1.41s/it]
 58%|█████▊    | 5758/10000 [39:27<1:13:46,  1.04s/it]
 58%|█████▊    | 5759/10000 [39:27<55:19,  1.28it/s]  
 58%|█████▊    | 5760/10000 [39:28<42:33,  1.66it/s]
                                                    
{'loss': 0.0793, 'grad_norm': 0.695988655090332, 'learning_rate': 4.162352447426177e-05}

 58%|█████▊    | 5760/10000 [39:28<42:33,  1.66it/s]
 58%|█████▊    | 5761/10000 [39:28<34:02,  2.08it/s]
 58%|█████▊    | 5762/10000 [39:28<27:43,  2.55it/s]
 58%|█████▊    | 5763/10000 [39:28<23:16,  3.03it/s]
 58%|█████▊    | 5764/10000 [39:28<20:11,  3.50it/s]
 58%|█████▊    | 5765/10000 [39:29<17:50,  3.96it/s]
 58%|█████▊    | 5766/10000 [39:29<16:11,  4.36it/s]
 58%|█████▊    | 5767/10000 [39:29<15:07,  4.67it/s]
 58%|█████▊    | 5768/10000 [39:29<14:19,  4.92it/s]
 58%|█████▊    | 5769/10000 [39:29<13:43,  5.14it/s]
 58%|█████▊    | 5770/10000 [39:29<13:34,  5.19it/s]
                                                    
{'loss': 0.0956, 'grad_norm': 0.748550295829773, 'learning_rate': 4.146056043021135e-05}

 58%|█████▊    | 5770/10000 [39:30<13:34,  5.19it/s]
 58%|█████▊    | 5771/10000 [39:30<13:38,  5.17it/s]
 58%|█████▊    | 5772/10000 [39:30<13:27,  5.23it/s]
 58%|█████▊    | 5773/10000 [39:30<13:19,  5.28it/s]
 58%|█████▊    | 5774/10000 [39:30<13:10,  5.35it/s]
 58%|█████▊    | 5775/10000 [39:30<13:49,  5.09it/s]
 58%|█████▊    | 5776/10000 [39:31<14:10,  4.96it/s]
 58%|█████▊    | 5777/10000 [39:31<14:33,  4.83it/s]
 58%|█████▊    | 5778/10000 [39:31<14:37,  4.81it/s]
 58%|█████▊    | 5779/10000 [39:31<14:42,  4.78it/s]
 58%|█████▊    | 5780/10000 [39:32<15:07,  4.65it/s]
                                                    
{'loss': 0.0943, 'grad_norm': 0.7856970429420471, 'learning_rate': 4.1297689772103944e-05}

 58%|█████▊    | 5780/10000 [39:32<15:07,  4.65it/s]
 58%|█████▊    | 5781/10000 [39:32<15:22,  4.58it/s]
 58%|█████▊    | 5782/10000 [39:32<15:04,  4.66it/s]
 58%|█████▊    | 5783/10000 [39:32<15:13,  4.62it/s]
 58%|█████▊    | 5784/10000 [39:32<15:02,  4.67it/s]
 58%|█████▊    | 5785/10000 [39:33<14:51,  4.73it/s]
 58%|█████▊    | 5786/10000 [39:33<14:07,  4.97it/s]
 58%|█████▊    | 5787/10000 [39:33<13:58,  5.02it/s]
 58%|█████▊    | 5788/10000 [39:33<13:45,  5.10it/s]
 58%|█████▊    | 5789/10000 [39:33<13:43,  5.12it/s]
 58%|█████▊    | 5790/10000 [39:34<13:27,  5.21it/s]
                                                    
{'loss': 0.0787, 'grad_norm': 0.7928652763366699, 'learning_rate': 4.113491428106694e-05}

 58%|█████▊    | 5790/10000 [39:34<13:27,  5.21it/s]
 58%|█████▊    | 5791/10000 [39:34<13:21,  5.25it/s]
 58%|█████▊    | 5792/10000 [39:34<13:08,  5.34it/s]
 58%|█████▊    | 5793/10000 [39:34<13:13,  5.30it/s]
 58%|█████▊    | 5794/10000 [39:34<13:10,  5.32it/s]
 58%|█████▊    | 5795/10000 [39:34<13:09,  5.33it/s]
 58%|█████▊    | 5796/10000 [39:35<13:18,  5.26it/s]
 58%|█████▊    | 5797/10000 [39:35<13:24,  5.22it/s]
 58%|█████▊    | 5798/10000 [39:35<13:16,  5.27it/s]
 58%|█████▊    | 5799/10000 [39:35<13:39,  5.13it/s]
 58%|█████▊    | 5800/10000 [39:35<13:51,  5.05it/s]
                                                    
{'loss': 0.081, 'grad_norm': 0.7445163130760193, 'learning_rate': 4.0972235737187055e-05}

 58%|█████▊    | 5800/10000 [39:36<13:51,  5.05it/s]
 58%|█████▊    | 5801/10000 [39:36<14:13,  4.92it/s]
 58%|█████▊    | 5802/10000 [39:36<14:06,  4.96it/s]
 58%|█████▊    | 5803/10000 [39:36<13:47,  5.07it/s]
 58%|█████▊    | 5804/10000 [39:36<13:31,  5.17it/s]
 58%|█████▊    | 5805/10000 [39:36<13:10,  5.30it/s]
 58%|█████▊    | 5806/10000 [39:37<12:54,  5.41it/s]
 58%|█████▊    | 5807/10000 [39:37<13:11,  5.30it/s]
 58%|█████▊    | 5808/10000 [39:37<13:59,  5.00it/s]
 58%|█████▊    | 5809/10000 [39:37<14:03,  4.97it/s]
 58%|█████▊    | 5810/10000 [39:37<14:05,  4.95it/s]
                                                    
{'loss': 0.0944, 'grad_norm': 0.9277859330177307, 'learning_rate': 4.080965591949076e-05}

 58%|█████▊    | 5810/10000 [39:37<14:05,  4.95it/s]
 58%|█████▊    | 5811/10000 [39:38<13:39,  5.11it/s]
 58%|█████▊    | 5812/10000 [39:38<13:07,  5.32it/s]
 58%|█████▊    | 5813/10000 [39:38<12:49,  5.44it/s]
 58%|█████▊    | 5814/10000 [39:38<13:00,  5.36it/s]
 58%|█████▊    | 5815/10000 [39:38<13:00,  5.36it/s]
 58%|█████▊    | 5816/10000 [39:39<13:29,  5.17it/s]
 58%|█████▊    | 5817/10000 [39:39<13:33,  5.14it/s]
 58%|█████▊    | 5818/10000 [39:39<13:52,  5.02it/s]
 58%|█████▊    | 5819/10000 [39:39<13:58,  4.99it/s]
 58%|█████▊    | 5820/10000 [39:39<13:57,  4.99it/s]
                                                    
{'loss': 0.0882, 'grad_norm': 0.7504689693450928, 'learning_rate': 4.0647176605924924e-05}

 58%|█████▊    | 5820/10000 [39:39<13:57,  4.99it/s]
 58%|█████▊    | 5821/10000 [39:40<13:44,  5.07it/s]
 58%|█████▊    | 5822/10000 [39:40<13:24,  5.19it/s]
 58%|█████▊    | 5823/10000 [39:40<13:01,  5.34it/s]
 58%|█████▊    | 5824/10000 [39:40<12:50,  5.42it/s]
 58%|█████▊    | 5825/10000 [39:40<12:55,  5.38it/s]
 58%|█████▊    | 5826/10000 [39:40<12:59,  5.36it/s]
 58%|█████▊    | 5827/10000 [39:41<12:55,  5.38it/s]
 58%|█████▊    | 5828/10000 [39:41<13:09,  5.28it/s]
 58%|█████▊    | 5829/10000 [39:41<13:09,  5.28it/s]
 58%|█████▊    | 5830/10000 [39:41<13:05,  5.31it/s]
                                                    
{'loss': 0.0849, 'grad_norm': 0.7360885143280029, 'learning_rate': 4.0484799573337255e-05}

 58%|█████▊    | 5830/10000 [39:41<13:05,  5.31it/s]
 58%|█████▊    | 5831/10000 [39:41<13:05,  5.31it/s]
 58%|█████▊    | 5832/10000 [39:42<12:54,  5.38it/s]
 58%|█████▊    | 5833/10000 [39:42<12:39,  5.49it/s]
 58%|█████▊    | 5834/10000 [39:42<12:37,  5.50it/s]
 58%|█████▊    | 5835/10000 [39:42<12:32,  5.53it/s]
 58%|█████▊    | 5836/10000 [39:42<12:26,  5.58it/s]
 58%|█████▊    | 5837/10000 [39:42<12:35,  5.51it/s]
 58%|█████▊    | 5838/10000 [39:43<12:37,  5.49it/s]
 58%|█████▊    | 5839/10000 [39:43<12:47,  5.42it/s]
 58%|█████▊    | 5840/10000 [39:43<12:47,  5.42it/s]
                                                    
{'loss': 0.0738, 'grad_norm': 0.856235682964325, 'learning_rate': 4.032252659745699e-05}

 58%|█████▊    | 5840/10000 [39:43<12:47,  5.42it/s]
 58%|█████▊    | 5841/10000 [39:43<13:00,  5.33it/s]
 58%|█████▊    | 5842/10000 [39:43<12:50,  5.40it/s]
 58%|█████▊    | 5843/10000 [39:44<12:48,  5.41it/s]
 58%|█████▊    | 5844/10000 [39:44<12:49,  5.40it/s]
 58%|█████▊    | 5845/10000 [39:44<12:49,  5.40it/s]
 58%|█████▊    | 5846/10000 [39:44<12:58,  5.33it/s]
 58%|█████▊    | 5847/10000 [39:44<12:59,  5.33it/s]
 58%|█████▊    | 5848/10000 [39:45<12:44,  5.43it/s]
 58%|█████▊    | 5849/10000 [39:45<12:40,  5.46it/s]
 58%|█████▊    | 5850/10000 [39:45<12:39,  5.47it/s]
                                                    
{'loss': 0.1007, 'grad_norm': 0.8238334059715271, 'learning_rate': 4.016035945287539e-05}

 58%|█████▊    | 5850/10000 [39:45<12:39,  5.47it/s]
 59%|█████▊    | 5851/10000 [39:45<13:08,  5.26it/s]
 59%|█████▊    | 5852/10000 [39:45<15:40,  4.41it/s]
 59%|█████▊    | 5853/10000 [39:46<14:48,  4.67it/s]
 59%|█████▊    | 5854/10000 [39:46<14:19,  4.82it/s]
 59%|█████▊    | 5855/10000 [39:46<13:40,  5.05it/s]
 59%|█████▊    | 5856/10000 [39:46<13:09,  5.25it/s]
 59%|█████▊    | 5857/10000 [39:46<13:04,  5.28it/s]
 59%|█████▊    | 5858/10000 [39:47<13:10,  5.24it/s]
 59%|█████▊    | 5859/10000 [39:47<13:11,  5.23it/s]
 59%|█████▊    | 5860/10000 [39:47<13:22,  5.16it/s]
                                                    
{'loss': 0.0866, 'grad_norm': 0.9030022621154785, 'learning_rate': 3.999829991302635e-05}

 59%|█████▊    | 5860/10000 [39:47<13:22,  5.16it/s]
 59%|█████▊    | 5861/10000 [39:47<13:50,  4.98it/s]
 59%|█████▊    | 5862/10000 [39:47<13:31,  5.10it/s]
 59%|█████▊    | 5863/10000 [39:47<13:16,  5.19it/s]
 59%|█████▊    | 5864/10000 [39:48<12:57,  5.32it/s]
 59%|█████▊    | 5865/10000 [39:48<12:38,  5.45it/s]
 59%|█████▊    | 5866/10000 [39:48<12:23,  5.56it/s]
 59%|█████▊    | 5867/10000 [39:48<12:20,  5.58it/s]
 59%|█████▊    | 5868/10000 [39:48<12:28,  5.52it/s]
 59%|█████▊    | 5869/10000 [39:49<12:31,  5.50it/s]
 59%|█████▊    | 5870/10000 [39:49<12:37,  5.45it/s]
                                                    
{'loss': 0.087, 'grad_norm': 0.7861736416816711, 'learning_rate': 3.983634975016707e-05}

 59%|█████▊    | 5870/10000 [39:49<12:37,  5.45it/s]
 59%|█████▊    | 5871/10000 [39:49<12:52,  5.35it/s]
 59%|█████▊    | 5872/10000 [39:49<12:38,  5.44it/s]
 59%|█████▊    | 5873/10000 [39:49<12:30,  5.50it/s]
 59%|█████▊    | 5874/10000 [39:49<12:16,  5.60it/s]
 59%|█████▉    | 5875/10000 [39:50<12:12,  5.63it/s]
 59%|█████▉    | 5876/10000 [39:50<12:12,  5.63it/s]
 59%|█████▉    | 5877/10000 [39:50<12:07,  5.67it/s]
 59%|█████▉    | 5878/10000 [39:50<12:05,  5.68it/s]
 59%|█████▉    | 5879/10000 [39:50<12:20,  5.56it/s]
 59%|█████▉    | 5880/10000 [39:51<12:21,  5.56it/s]
                                                    
{'loss': 0.0968, 'grad_norm': 0.5808513164520264, 'learning_rate': 3.967451073535854e-05}

 59%|█████▉    | 5880/10000 [39:51<12:21,  5.56it/s]
 59%|█████▉    | 5881/10000 [39:51<12:26,  5.52it/s]
 59%|█████▉    | 5882/10000 [39:51<12:22,  5.55it/s]
 59%|█████▉    | 5883/10000 [39:51<12:13,  5.62it/s]
 59%|█████▉    | 5884/10000 [39:51<12:05,  5.67it/s]
 59%|█████▉    | 5885/10000 [39:51<12:04,  5.68it/s]
 59%|█████▉    | 5886/10000 [39:52<12:09,  5.64it/s]
 59%|█████▉    | 5887/10000 [39:52<12:05,  5.67it/s]
 59%|█████▉    | 5888/10000 [39:52<12:05,  5.67it/s]
 59%|█████▉    | 5889/10000 [39:52<12:14,  5.60it/s]
 59%|█████▉    | 5890/10000 [39:52<12:07,  5.65it/s]
                                                    
{'loss': 0.0815, 'grad_norm': 0.7054378390312195, 'learning_rate': 3.951278463844633e-05}

 59%|█████▉    | 5890/10000 [39:52<12:07,  5.65it/s]
 59%|█████▉    | 5891/10000 [39:52<12:25,  5.51it/s]
 59%|█████▉    | 5892/10000 [39:53<12:26,  5.50it/s]
 59%|█████▉    | 5893/10000 [39:53<12:27,  5.49it/s]
 59%|█████▉    | 5894/10000 [39:53<12:25,  5.51it/s]
 59%|█████▉    | 5895/10000 [39:53<12:29,  5.48it/s]
 59%|█████▉    | 5896/10000 [39:53<12:25,  5.50it/s]
 59%|█████▉    | 5897/10000 [39:54<12:21,  5.53it/s]
 59%|█████▉    | 5898/10000 [39:54<12:29,  5.47it/s]
 59%|█████▉    | 5899/10000 [39:54<12:26,  5.49it/s]
 59%|█████▉    | 5900/10000 [39:54<12:34,  5.43it/s]
                                                    
{'loss': 0.0807, 'grad_norm': 0.6949955224990845, 'learning_rate': 3.935117322804111e-05}

 59%|█████▉    | 5900/10000 [39:54<12:34,  5.43it/s]
 59%|█████▉    | 5901/10000 [39:54<12:47,  5.34it/s]
 59%|█████▉    | 5902/10000 [39:55<12:41,  5.38it/s]
 59%|█████▉    | 5903/10000 [39:55<12:32,  5.45it/s]
 59%|█████▉    | 5904/10000 [39:55<12:24,  5.50it/s]
 59%|█████▉    | 5905/10000 [39:55<12:15,  5.56it/s]
 59%|█████▉    | 5906/10000 [39:55<12:14,  5.57it/s]
 59%|█████▉    | 5907/10000 [39:55<12:13,  5.58it/s]
 59%|█████▉    | 5908/10000 [39:56<12:37,  5.41it/s]
 59%|█████▉    | 5909/10000 [39:56<12:59,  5.25it/s]
 59%|█████▉    | 5910/10000 [39:56<12:53,  5.29it/s]
                                                    
{'loss': 0.0759, 'grad_norm': 0.7594128847122192, 'learning_rate': 3.918967827149938e-05}

 59%|█████▉    | 5910/10000 [39:56<12:53,  5.29it/s]
 59%|█████▉    | 5911/10000 [39:56<13:02,  5.23it/s]
 59%|█████▉    | 5912/10000 [39:56<13:32,  5.03it/s]
 59%|█████▉    | 5913/10000 [39:57<13:35,  5.01it/s]
 59%|█████▉    | 5914/10000 [39:57<13:44,  4.96it/s]
 59%|█████▉    | 5915/10000 [39:57<13:49,  4.93it/s]
 59%|█████▉    | 5916/10000 [39:57<13:35,  5.01it/s]
 59%|█████▉    | 5917/10000 [39:57<13:16,  5.12it/s]
 59%|█████▉    | 5918/10000 [39:58<13:20,  5.10it/s]
 59%|█████▉    | 5919/10000 [39:58<13:14,  5.14it/s]
 59%|█████▉    | 5920/10000 [39:58<13:32,  5.02it/s]
                                                    
{'loss': 0.1058, 'grad_norm': 0.8318119645118713, 'learning_rate': 3.9028301534904094e-05}

 59%|█████▉    | 5920/10000 [39:58<13:32,  5.02it/s]
 59%|█████▉    | 5921/10000 [39:58<14:02,  4.84it/s]
 59%|█████▉    | 5922/10000 [39:58<13:29,  5.04it/s]
 59%|█████▉    | 5923/10000 [39:59<13:05,  5.19it/s]
 59%|█████▉    | 5924/10000 [39:59<12:52,  5.28it/s]
 59%|█████▉    | 5925/10000 [39:59<12:36,  5.39it/s]
 59%|█████▉    | 5926/10000 [39:59<12:37,  5.38it/s]
 59%|█████▉    | 5927/10000 [39:59<12:50,  5.28it/s]
 59%|█████▉    | 5928/10000 [40:00<13:53,  4.88it/s]
 59%|█████▉    | 5929/10000 [40:00<13:45,  4.93it/s]Rank 0, Worker 1: Wait for shard 68 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 59%|█████▉    | 5930/10000 [40:00<13:47,  4.92it/s]
                                                    
{'loss': 0.0877, 'grad_norm': 0.6270005106925964, 'learning_rate': 3.88670447830454e-05}

 59%|█████▉    | 5930/10000 [40:00<13:47,  4.92it/s]
 59%|█████▉    | 5931/10000 [40:00<13:48,  4.91it/s]
 59%|█████▉    | 5932/10000 [40:00<13:32,  5.01it/s]
 59%|█████▉    | 5933/10000 [40:01<13:24,  5.05it/s]
 59%|█████▉    | 5934/10000 [40:01<13:27,  5.04it/s]
 59%|█████▉    | 5935/10000 [40:01<13:28,  5.03it/s]
 59%|█████▉    | 5936/10000 [40:01<13:17,  5.10it/s]
 59%|█████▉    | 5937/10000 [40:01<13:05,  5.17it/s]
 59%|█████▉    | 5938/10000 [40:02<13:40,  4.95it/s]
 59%|█████▉    | 5939/10000 [40:02<14:15,  4.75it/s]
 59%|█████▉    | 5940/10000 [40:02<14:39,  4.62it/s]
                                                    
{'loss': 0.0933, 'grad_norm': 0.7320938110351562, 'learning_rate': 3.870590977940132e-05}

 59%|█████▉    | 5940/10000 [40:02<14:39,  4.62it/s]
 59%|█████▉    | 5941/10000 [40:02<15:39,  4.32it/s]
 59%|█████▉    | 5942/10000 [40:03<15:56,  4.24it/s]
 59%|█████▉    | 5943/10000 [40:03<15:27,  4.38it/s]
 59%|█████▉    | 5944/10000 [40:03<14:54,  4.53it/s]
 59%|█████▉    | 5945/10000 [40:03<14:18,  4.72it/s]
 59%|█████▉    | 5946/10000 [40:03<13:53,  4.86it/s]
 59%|█████▉    | 5947/10000 [40:04<14:10,  4.77it/s]
 59%|█████▉    | 5948/10000 [40:04<14:43,  4.58it/s]
 59%|█████▉    | 5949/10000 [40:04<14:54,  4.53it/s]
 60%|█████▉    | 5950/10000 [40:04<14:48,  4.56it/s]
                                                    
{'loss': 0.0785, 'grad_norm': 0.6579483151435852, 'learning_rate': 3.8544898286118404e-05}

 60%|█████▉    | 5950/10000 [40:04<14:48,  4.56it/s]
 60%|█████▉    | 5951/10000 [40:04<14:43,  4.58it/s]
 60%|█████▉    | 5952/10000 [40:05<14:09,  4.76it/s]
 60%|█████▉    | 5953/10000 [40:05<13:44,  4.91it/s]
 60%|█████▉    | 5954/10000 [40:05<13:52,  4.86it/s]
 60%|█████▉    | 5955/10000 [40:05<13:50,  4.87it/s]
 60%|█████▉    | 5956/10000 [40:05<13:46,  4.89it/s]
 60%|█████▉    | 5957/10000 [40:06<14:07,  4.77it/s]
 60%|█████▉    | 5958/10000 [40:06<14:32,  4.63it/s]
 60%|█████▉    | 5959/10000 [40:06<14:36,  4.61it/s]
 60%|█████▉    | 5960/10000 [40:06<14:41,  4.58it/s]
                                                    
{'loss': 0.1013, 'grad_norm': 0.6441275477409363, 'learning_rate': 3.838401206399257e-05}

 60%|█████▉    | 5960/10000 [40:06<14:41,  4.58it/s]
 60%|█████▉    | 5961/10000 [40:07<14:56,  4.50it/s]
 60%|█████▉    | 5962/10000 [40:07<14:24,  4.67it/s]
 60%|█████▉    | 5963/10000 [40:07<13:57,  4.82it/s]
 60%|█████▉    | 5964/10000 [40:07<13:38,  4.93it/s]
 60%|█████▉    | 5965/10000 [40:07<13:37,  4.94it/s]
 60%|█████▉    | 5966/10000 [40:08<13:41,  4.91it/s]
 60%|█████▉    | 5967/10000 [40:08<14:03,  4.78it/s]
 60%|█████▉    | 5968/10000 [40:08<14:31,  4.63it/s]
 60%|█████▉    | 5969/10000 [40:08<14:48,  4.54it/s]
 60%|█████▉    | 5970/10000 [40:08<14:35,  4.60it/s]
                                                    
{'loss': 0.0738, 'grad_norm': 0.8163285255432129, 'learning_rate': 3.822325287244975e-05}

 60%|█████▉    | 5970/10000 [40:08<14:35,  4.60it/s]
 60%|█████▉    | 5971/10000 [40:09<14:48,  4.53it/s]
 60%|█████▉    | 5972/10000 [40:09<14:14,  4.71it/s]
 60%|█████▉    | 5973/10000 [40:09<13:55,  4.82it/s]
 60%|█████▉    | 5974/10000 [40:09<14:02,  4.78it/s]
 60%|█████▉    | 5975/10000 [40:10<14:28,  4.64it/s]
 60%|█████▉    | 5976/10000 [40:10<14:42,  4.56it/s]
 60%|█████▉    | 5977/10000 [40:10<15:22,  4.36it/s]
 60%|█████▉    | 5978/10000 [40:10<15:12,  4.41it/s]
 60%|█████▉    | 5979/10000 [40:10<14:31,  4.61it/s]
 60%|█████▉    | 5980/10000 [40:11<14:00,  4.78it/s]
                                                    
{'loss': 0.0861, 'grad_norm': 0.830970048904419, 'learning_rate': 3.8062622469526725e-05}

 60%|█████▉    | 5980/10000 [40:11<14:00,  4.78it/s]
 60%|█████▉    | 5981/10000 [40:11<13:56,  4.81it/s]
 60%|█████▉    | 5982/10000 [40:11<13:53,  4.82it/s]
 60%|█████▉    | 5983/10000 [40:11<14:22,  4.66it/s]
 60%|█████▉    | 5984/10000 [40:11<14:42,  4.55it/s]
 60%|█████▉    | 5985/10000 [40:12<15:20,  4.36it/s]
 60%|█████▉    | 5986/10000 [40:12<16:03,  4.17it/s]
 60%|█████▉    | 5987/10000 [40:12<15:15,  4.38it/s]
 60%|█████▉    | 5988/10000 [40:12<14:27,  4.62it/s]
 60%|█████▉    | 5989/10000 [40:13<14:03,  4.75it/s]
 60%|█████▉    | 5990/10000 [40:13<13:49,  4.83it/s]
                                                    
{'loss': 0.0898, 'grad_norm': 0.9203442335128784, 'learning_rate': 3.790212261185183e-05}

 60%|█████▉    | 5990/10000 [40:13<13:49,  4.83it/s]
 60%|█████▉    | 5991/10000 [40:13<14:13,  4.70it/s]
 60%|█████▉    | 5992/10000 [40:13<14:33,  4.59it/s]
 60%|█████▉    | 5993/10000 [40:13<14:35,  4.58it/s]
 60%|█████▉    | 5994/10000 [40:14<14:47,  4.52it/s]
 60%|█████▉    | 5995/10000 [40:14<14:21,  4.65it/s]
 60%|█████▉    | 5996/10000 [40:14<13:59,  4.77it/s]
 60%|█████▉    | 5997/10000 [40:14<13:37,  4.90it/s]
 60%|█████▉    | 5998/10000 [40:14<13:36,  4.90it/s]
 60%|█████▉    | 5999/10000 [40:15<14:10,  4.70it/s]
 60%|██████    | 6000/10000 [40:15<14:16,  4.67it/s]Rank 0, Worker 0: Caching shard...Rank 0, Worker 1: Caching shard...

Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.60 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.11 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.13 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.28 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.57 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.73 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 6000: eval/loss = 0.1021 (37s)

                                                    
{'loss': 0.1023, 'grad_norm': 0.8261857628822327, 'learning_rate': 3.7741755054625794e-05}

 60%|██████    | 6000/10000 [40:52<14:16,  4.67it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-6000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-6000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-6000/wandb_config.json

 60%|██████    | 6001/10000 [41:20<21:49:14, 19.64s/it]
 60%|██████    | 6002/10000 [41:20<15:19:44, 13.80s/it]
 60%|██████    | 6003/10000 [41:20<10:47:06,  9.71s/it]
 60%|██████    | 6004/10000 [41:20<7:36:22,  6.85s/it] 
 60%|██████    | 6005/10000 [41:21<5:23:00,  4.85s/it]
 60%|██████    | 6006/10000 [41:21<3:49:33,  3.45s/it]
 60%|██████    | 6007/10000 [41:21<2:44:23,  2.47s/it]
 60%|██████    | 6008/10000 [41:21<1:59:01,  1.79s/it]
 60%|██████    | 6009/10000 [41:21<1:26:55,  1.31s/it]
 60%|██████    | 6010/10000 [41:22<1:04:31,  1.03it/s]
                                                      
{'loss': 0.0851, 'grad_norm': 0.7447966933250427, 'learning_rate': 3.758152155160255e-05}

 60%|██████    | 6010/10000 [41:22<1:04:31,  1.03it/s]
 60%|██████    | 6011/10000 [41:22<49:08,  1.35it/s]  
 60%|██████    | 6012/10000 [41:22<38:05,  1.74it/s]
 60%|██████    | 6013/10000 [41:22<30:13,  2.20it/s]
 60%|██████    | 6014/10000 [41:22<25:28,  2.61it/s]
 60%|██████    | 6015/10000 [41:23<22:33,  2.94it/s]
 60%|██████    | 6016/10000 [41:23<19:39,  3.38it/s]
 60%|██████    | 6017/10000 [41:23<17:50,  3.72it/s]
 60%|██████    | 6018/10000 [41:23<16:48,  3.95it/s]
 60%|██████    | 6019/10000 [41:23<15:47,  4.20it/s]
 60%|██████    | 6020/10000 [41:24<14:56,  4.44it/s]
                                                    
{'loss': 0.0922, 'grad_norm': 0.7559837102890015, 'learning_rate': 3.742142385506999e-05}

 60%|██████    | 6020/10000 [41:24<14:56,  4.44it/s]
 60%|██████    | 6021/10000 [41:24<14:27,  4.59it/s]
 60%|██████    | 6022/10000 [41:24<14:03,  4.72it/s]
 60%|██████    | 6023/10000 [41:24<13:41,  4.84it/s]
 60%|██████    | 6024/10000 [41:24<13:30,  4.90it/s]
 60%|██████    | 6025/10000 [41:25<13:26,  4.93it/s]
 60%|██████    | 6026/10000 [41:25<13:24,  4.94it/s]
 60%|██████    | 6027/10000 [41:25<13:18,  4.97it/s]
 60%|██████    | 6028/10000 [41:25<13:14,  5.00it/s]
 60%|██████    | 6029/10000 [41:25<12:59,  5.09it/s]
 60%|██████    | 6030/10000 [41:26<12:45,  5.19it/s]
                                                    
{'loss': 0.0823, 'grad_norm': 0.912108838558197, 'learning_rate': 3.72614637158309e-05}

 60%|██████    | 6030/10000 [41:26<12:45,  5.19it/s]
 60%|██████    | 6031/10000 [41:26<12:54,  5.13it/s]
 60%|██████    | 6032/10000 [41:26<12:54,  5.12it/s]
 60%|██████    | 6033/10000 [41:26<13:01,  5.08it/s]
 60%|██████    | 6034/10000 [41:26<13:28,  4.91it/s]
 60%|██████    | 6035/10000 [41:27<14:06,  4.68it/s]
 60%|██████    | 6036/10000 [41:27<13:40,  4.83it/s]
 60%|██████    | 6037/10000 [41:27<13:55,  4.75it/s]
 60%|██████    | 6038/10000 [41:27<14:29,  4.56it/s]
 60%|██████    | 6039/10000 [41:27<14:35,  4.53it/s]
 60%|██████    | 6040/10000 [41:28<14:40,  4.50it/s]
                                                    
{'loss': 0.09, 'grad_norm': 0.7563724517822266, 'learning_rate': 3.710164288318371e-05}

 60%|██████    | 6040/10000 [41:28<14:40,  4.50it/s]
 60%|██████    | 6041/10000 [41:28<14:57,  4.41it/s]
 60%|██████    | 6042/10000 [41:28<15:05,  4.37it/s]
 60%|██████    | 6043/10000 [41:28<14:05,  4.68it/s]
 60%|██████    | 6044/10000 [41:28<13:19,  4.95it/s]
 60%|██████    | 6045/10000 [41:29<13:09,  5.01it/s]
 60%|██████    | 6046/10000 [41:29<12:59,  5.07it/s]
 60%|██████    | 6047/10000 [41:29<13:08,  5.01it/s]
 60%|██████    | 6048/10000 [41:29<13:47,  4.78it/s]
 60%|██████    | 6049/10000 [41:30<13:49,  4.76it/s]
 60%|██████    | 6050/10000 [41:30<13:53,  4.74it/s]
                                                    
{'loss': 0.0877, 'grad_norm': 0.7515089511871338, 'learning_rate': 3.694196310490345e-05}

 60%|██████    | 6050/10000 [41:30<13:53,  4.74it/s]
 61%|██████    | 6051/10000 [41:30<13:59,  4.71it/s]
 61%|██████    | 6052/10000 [41:30<13:43,  4.79it/s]
 61%|██████    | 6053/10000 [41:30<13:11,  4.99it/s]
 61%|██████    | 6054/10000 [41:31<12:43,  5.17it/s]
 61%|██████    | 6055/10000 [41:31<12:28,  5.27it/s]
 61%|██████    | 6056/10000 [41:31<12:23,  5.30it/s]
 61%|██████    | 6057/10000 [41:31<12:30,  5.25it/s]
 61%|██████    | 6058/10000 [41:31<13:01,  5.04it/s]
 61%|██████    | 6059/10000 [41:32<13:29,  4.87it/s]
 61%|██████    | 6060/10000 [41:32<13:36,  4.83it/s]
                                                    
{'loss': 0.0777, 'grad_norm': 0.8701909184455872, 'learning_rate': 3.678242612722259e-05}

 61%|██████    | 6060/10000 [41:32<13:36,  4.83it/s]
 61%|██████    | 6061/10000 [41:32<13:57,  4.70it/s]
 61%|██████    | 6062/10000 [41:32<13:32,  4.84it/s]
 61%|██████    | 6063/10000 [41:32<13:01,  5.04it/s]
 61%|██████    | 6064/10000 [41:33<12:41,  5.17it/s]
 61%|██████    | 6065/10000 [41:33<12:34,  5.22it/s]
 61%|██████    | 6066/10000 [41:33<12:59,  5.04it/s]
 61%|██████    | 6067/10000 [41:33<13:05,  5.01it/s]
 61%|██████    | 6068/10000 [41:33<13:39,  4.80it/s]
 61%|██████    | 6069/10000 [41:34<14:14,  4.60it/s]
 61%|██████    | 6070/10000 [41:34<14:35,  4.49it/s]
                                                    
{'loss': 0.0691, 'grad_norm': 0.923819899559021, 'learning_rate': 3.6623033694811953e-05}

 61%|██████    | 6070/10000 [41:34<14:35,  4.49it/s]
 61%|██████    | 6071/10000 [41:34<14:27,  4.53it/s]
 61%|██████    | 6072/10000 [41:34<13:37,  4.80it/s]
 61%|██████    | 6073/10000 [41:34<13:14,  4.95it/s]
 61%|██████    | 6074/10000 [41:35<13:02,  5.02it/s]
 61%|██████    | 6075/10000 [41:35<13:19,  4.91it/s]
 61%|██████    | 6076/10000 [41:35<13:53,  4.71it/s]
 61%|██████    | 6077/10000 [41:35<14:21,  4.55it/s]
 61%|██████    | 6078/10000 [41:36<14:41,  4.45it/s]
 61%|██████    | 6079/10000 [41:36<14:13,  4.59it/s]
 61%|██████    | 6080/10000 [41:36<13:23,  4.88it/s]
                                                    
{'loss': 0.0856, 'grad_norm': 0.8755983114242554, 'learning_rate': 3.6463787550761665e-05}

 61%|██████    | 6080/10000 [41:36<13:23,  4.88it/s]Rank 0, Worker 2: Wait for shard 66 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 61%|██████    | 6081/10000 [41:36<13:15,  4.92it/s]
 61%|██████    | 6082/10000 [41:36<13:11,  4.95it/s]
 61%|██████    | 6083/10000 [41:37<13:43,  4.76it/s]
 61%|██████    | 6084/10000 [41:37<14:37,  4.46it/s]
 61%|██████    | 6085/10000 [41:37<14:44,  4.43it/s]
 61%|██████    | 6086/10000 [41:37<14:41,  4.44it/s]
 61%|██████    | 6087/10000 [41:37<14:19,  4.55it/s]
 61%|██████    | 6088/10000 [41:38<13:57,  4.67it/s]Rank 0, Worker 4: Wait for shard 43 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 61%|██████    | 6089/10000 [41:38<14:13,  4.58it/s]
 61%|██████    | 6090/10000 [41:38<14:29,  4.50it/s]
                                                    
{'loss': 0.093, 'grad_norm': 0.8513551950454712, 'learning_rate': 3.630468943656202e-05}

 61%|██████    | 6090/10000 [41:38<14:29,  4.50it/s]
 61%|██████    | 6091/10000 [41:38<14:57,  4.36it/s]
 61%|██████    | 6092/10000 [41:39<14:56,  4.36it/s]
 61%|██████    | 6093/10000 [41:39<14:54,  4.37it/s]
 61%|██████    | 6094/10000 [41:39<14:39,  4.44it/s]
 61%|██████    | 6095/10000 [41:39<13:54,  4.68it/s]
 61%|██████    | 6096/10000 [41:39<13:44,  4.74it/s]
 61%|██████    | 6097/10000 [41:40<14:06,  4.61it/s]
 61%|██████    | 6098/10000 [41:40<14:30,  4.48it/s]
 61%|██████    | 6099/10000 [41:40<14:45,  4.41it/s]
 61%|██████    | 6100/10000 [41:40<15:18,  4.25it/s]
                                                    
{'loss': 0.0848, 'grad_norm': 0.8277741074562073, 'learning_rate': 3.6145741092084523e-05}

 61%|██████    | 6100/10000 [41:40<15:18,  4.25it/s]
 61%|██████    | 6101/10000 [41:41<15:19,  4.24it/s]Rank 0, Worker 5: Wait for shard 76 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 61%|██████    | 6102/10000 [41:41<15:08,  4.29it/s]
 61%|██████    | 6103/10000 [41:41<14:58,  4.33it/s]
 61%|██████    | 6104/10000 [41:41<14:55,  4.35it/s]
 61%|██████    | 6105/10000 [41:42<15:02,  4.31it/s]
 61%|██████    | 6106/10000 [41:42<15:37,  4.15it/s]
 61%|██████    | 6107/10000 [41:42<15:40,  4.14it/s]
 61%|██████    | 6108/10000 [41:42<15:46,  4.11it/s]
 61%|██████    | 6109/10000 [41:42<15:16,  4.25it/s]
 61%|██████    | 6110/10000 [41:43<14:52,  4.36it/s]
                                                    
{'loss': 0.0874, 'grad_norm': 0.8102641701698303, 'learning_rate': 3.598694425556278e-05}

 61%|██████    | 6110/10000 [41:43<14:52,  4.36it/s]
 61%|██████    | 6111/10000 [41:43<14:44,  4.40it/s]Rank 0, Worker 3: Wait for shard 48 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 61%|██████    | 6112/10000 [41:43<14:31,  4.46it/s]
 61%|██████    | 6113/10000 [41:43<14:23,  4.50it/s]
 61%|██████    | 6114/10000 [41:44<14:24,  4.49it/s]
 61%|██████    | 6115/10000 [41:44<14:42,  4.40it/s]
 61%|██████    | 6116/10000 [41:44<14:44,  4.39it/s]
 61%|██████    | 6117/10000 [41:44<14:56,  4.33it/s]
 61%|██████    | 6118/10000 [41:45<15:14,  4.25it/s]
 61%|██████    | 6119/10000 [41:45<15:27,  4.18it/s]
 61%|██████    | 6120/10000 [41:45<15:13,  4.25it/s]
                                                    
{'loss': 0.0775, 'grad_norm': 0.7897539138793945, 'learning_rate': 3.58283006635736e-05}

 61%|██████    | 6120/10000 [41:45<15:13,  4.25it/s]
 61%|██████    | 6121/10000 [41:45<15:47,  4.10it/s]
 61%|██████    | 6122/10000 [41:46<15:30,  4.17it/s]
 61%|██████    | 6123/10000 [41:46<15:01,  4.30it/s]
 61%|██████    | 6124/10000 [41:46<14:37,  4.42it/s]
 61%|██████▏   | 6125/10000 [41:46<14:13,  4.54it/s]
 61%|██████▏   | 6126/10000 [41:46<14:09,  4.56it/s]
 61%|██████▏   | 6127/10000 [41:47<14:22,  4.49it/s]
 61%|██████▏   | 6128/10000 [41:47<14:02,  4.60it/s]
 61%|██████▏   | 6129/10000 [41:47<14:38,  4.41it/s]
 61%|██████▏   | 6130/10000 [41:47<15:08,  4.26it/s]
                                                    
{'loss': 0.0799, 'grad_norm': 0.7320510745048523, 'learning_rate': 3.566981205101781e-05}

 61%|██████▏   | 6130/10000 [41:47<15:08,  4.26it/s]
 61%|██████▏   | 6131/10000 [41:48<15:19,  4.21it/s]
 61%|██████▏   | 6132/10000 [41:48<15:19,  4.21it/s]
 61%|██████▏   | 6133/10000 [41:48<15:00,  4.30it/s]
 61%|██████▏   | 6134/10000 [41:48<14:27,  4.46it/s]
 61%|██████▏   | 6135/10000 [41:48<14:00,  4.60it/s]
 61%|██████▏   | 6136/10000 [41:49<14:00,  4.60it/s]
 61%|██████▏   | 6137/10000 [41:49<14:51,  4.33it/s]
 61%|██████▏   | 6138/10000 [41:49<15:05,  4.26it/s]
 61%|██████▏   | 6139/10000 [41:49<15:24,  4.18it/s]
 61%|██████▏   | 6140/10000 [41:50<15:38,  4.11it/s]
                                                    
{'loss': 0.0792, 'grad_norm': 0.7191510200500488, 'learning_rate': 3.5511480151101556e-05}

 61%|██████▏   | 6140/10000 [41:50<15:38,  4.11it/s]
 61%|██████▏   | 6141/10000 [41:50<15:47,  4.07it/s]
 61%|██████▏   | 6142/10000 [41:50<15:22,  4.18it/s]
 61%|██████▏   | 6143/10000 [41:50<15:17,  4.21it/s]
 61%|██████▏   | 6144/10000 [41:51<14:51,  4.32it/s]
 61%|██████▏   | 6145/10000 [41:51<14:44,  4.36it/s]
 61%|██████▏   | 6146/10000 [41:51<15:06,  4.25it/s]
 61%|██████▏   | 6147/10000 [41:51<15:03,  4.26it/s]
 61%|██████▏   | 6148/10000 [41:52<15:28,  4.15it/s]
 61%|██████▏   | 6149/10000 [41:52<15:30,  4.14it/s]
 62%|██████▏   | 6150/10000 [41:52<15:04,  4.26it/s]
                                                    
{'loss': 0.0721, 'grad_norm': 0.8049972057342529, 'learning_rate': 3.5353306695317104e-05}

 62%|██████▏   | 6150/10000 [41:52<15:04,  4.26it/s]
 62%|██████▏   | 6151/10000 [41:52<15:09,  4.23it/s]
 62%|██████▏   | 6152/10000 [41:52<14:50,  4.32it/s]
 62%|██████▏   | 6153/10000 [41:53<14:31,  4.41it/s]
 62%|██████▏   | 6154/10000 [41:53<14:21,  4.46it/s]
 62%|██████▏   | 6155/10000 [41:53<14:55,  4.29it/s]
 62%|██████▏   | 6156/10000 [41:53<14:56,  4.29it/s]Rank 0, Worker 0: Wait for shard 75 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 62%|██████▏   | 6157/10000 [41:54<15:03,  4.25it/s]
 62%|██████▏   | 6158/10000 [41:54<14:56,  4.29it/s]
 62%|██████▏   | 6159/10000 [41:54<14:33,  4.40it/s]
 62%|██████▏   | 6160/10000 [41:54<14:17,  4.48it/s]
                                                    
{'loss': 0.0811, 'grad_norm': 0.7906951904296875, 'learning_rate': 3.519529341342402e-05}

 62%|██████▏   | 6160/10000 [41:54<14:17,  4.48it/s]
 62%|██████▏   | 6161/10000 [41:55<14:55,  4.28it/s]
 62%|██████▏   | 6162/10000 [41:55<14:47,  4.32it/s]
 62%|██████▏   | 6163/10000 [41:55<15:06,  4.23it/s]
 62%|██████▏   | 6164/10000 [41:55<14:56,  4.28it/s]
 62%|██████▏   | 6165/10000 [41:55<15:21,  4.16it/s]
 62%|██████▏   | 6166/10000 [41:56<15:48,  4.04it/s]
 62%|██████▏   | 6167/10000 [41:56<15:12,  4.20it/s]
 62%|██████▏   | 6168/10000 [41:56<14:48,  4.32it/s]
 62%|██████▏   | 6169/10000 [41:56<14:23,  4.43it/s]
 62%|██████▏   | 6170/10000 [41:57<14:02,  4.55it/s]
                                                    
{'loss': 0.092, 'grad_norm': 0.6912961006164551, 'learning_rate': 3.503744203343026e-05}

 62%|██████▏   | 6170/10000 [41:57<14:02,  4.55it/s]
 62%|██████▏   | 6171/10000 [41:57<14:36,  4.37it/s]
 62%|██████▏   | 6172/10000 [41:57<14:33,  4.38it/s]
 62%|██████▏   | 6173/10000 [41:57<14:52,  4.29it/s]
 62%|██████▏   | 6174/10000 [41:58<16:24,  3.89it/s]
 62%|██████▏   | 6175/10000 [41:58<17:29,  3.65it/s]
 62%|██████▏   | 6176/10000 [41:58<17:37,  3.62it/s]
 62%|██████▏   | 6177/10000 [41:58<17:46,  3.59it/s]
 62%|██████▏   | 6178/10000 [41:59<17:30,  3.64it/s]
 62%|██████▏   | 6179/10000 [41:59<16:06,  3.95it/s]
 62%|██████▏   | 6180/10000 [41:59<15:16,  4.17it/s]
                                                    
{'loss': 0.0815, 'grad_norm': 0.7403696179389954, 'learning_rate': 3.487975428157318e-05}

 62%|██████▏   | 6180/10000 [41:59<15:16,  4.17it/s]
 62%|██████▏   | 6181/10000 [41:59<15:11,  4.19it/s]
 62%|██████▏   | 6182/10000 [42:00<15:25,  4.13it/s]
 62%|██████▏   | 6183/10000 [42:00<15:41,  4.05it/s]
 62%|██████▏   | 6184/10000 [42:00<16:35,  3.83it/s]
 62%|██████▏   | 6185/10000 [42:00<16:03,  3.96it/s]
 62%|██████▏   | 6186/10000 [42:01<15:51,  4.01it/s]
 62%|██████▏   | 6187/10000 [42:01<15:19,  4.15it/s]
 62%|██████▏   | 6188/10000 [42:01<14:36,  4.35it/s]
 62%|██████▏   | 6189/10000 [42:01<14:11,  4.47it/s]
 62%|██████▏   | 6190/10000 [42:02<13:56,  4.56it/s]
                                                    
{'loss': 0.0927, 'grad_norm': 0.7386904954910278, 'learning_rate': 3.472223188230083e-05}

 62%|██████▏   | 6190/10000 [42:02<13:56,  4.56it/s]
 62%|██████▏   | 6191/10000 [42:02<14:03,  4.52it/s]
 62%|██████▏   | 6192/10000 [42:02<13:47,  4.60it/s]
 62%|██████▏   | 6193/10000 [42:02<13:41,  4.63it/s]
 62%|██████▏   | 6194/10000 [42:02<14:44,  4.30it/s]
 62%|██████▏   | 6195/10000 [42:03<15:48,  4.01it/s]
 62%|██████▏   | 6196/10000 [42:03<15:25,  4.11it/s]
 62%|██████▏   | 6197/10000 [42:03<15:03,  4.21it/s]
 62%|██████▏   | 6198/10000 [42:03<14:08,  4.48it/s]
 62%|██████▏   | 6199/10000 [42:04<13:23,  4.73it/s]
 62%|██████▏   | 6200/10000 [42:04<13:10,  4.81it/s]
                                                    
{'loss': 0.0853, 'grad_norm': 0.6390708088874817, 'learning_rate': 3.4564876558252866e-05}

 62%|██████▏   | 6200/10000 [42:04<13:10,  4.81it/s]
 62%|██████▏   | 6201/10000 [42:04<13:11,  4.80it/s]
 62%|██████▏   | 6202/10000 [42:04<13:04,  4.84it/s]
 62%|██████▏   | 6203/10000 [42:04<13:06,  4.83it/s]
 62%|██████▏   | 6204/10000 [42:05<13:36,  4.65it/s]
 62%|██████▏   | 6205/10000 [42:05<13:45,  4.60it/s]
 62%|██████▏   | 6206/10000 [42:05<13:37,  4.64it/s]
 62%|██████▏   | 6207/10000 [42:05<13:18,  4.75it/s]
 62%|██████▏   | 6208/10000 [42:05<12:45,  4.96it/s]
 62%|██████▏   | 6209/10000 [42:06<12:23,  5.10it/s]
 62%|██████▏   | 6210/10000 [42:06<12:17,  5.14it/s]
                                                    
{'loss': 0.0828, 'grad_norm': 0.8252303004264832, 'learning_rate': 3.440769003024195e-05}

 62%|██████▏   | 6210/10000 [42:06<12:17,  5.14it/s]
 62%|██████▏   | 6211/10000 [42:06<12:37,  5.00it/s]
 62%|██████▏   | 6212/10000 [42:06<12:47,  4.94it/s]
 62%|██████▏   | 6213/10000 [42:06<12:58,  4.87it/s]
 62%|██████▏   | 6214/10000 [42:07<13:13,  4.77it/s]
 62%|██████▏   | 6215/10000 [42:07<12:59,  4.85it/s]
 62%|██████▏   | 6216/10000 [42:07<12:27,  5.06it/s]
 62%|██████▏   | 6217/10000 [42:07<12:00,  5.25it/s]
 62%|██████▏   | 6218/10000 [42:07<12:07,  5.20it/s]
 62%|██████▏   | 6219/10000 [42:08<12:30,  5.04it/s]
 62%|██████▏   | 6220/10000 [42:08<12:59,  4.85it/s]
                                                    
{'loss': 0.1133, 'grad_norm': 0.8028150200843811, 'learning_rate': 3.425067401723477e-05}

 62%|██████▏   | 6220/10000 [42:08<12:59,  4.85it/s]
 62%|██████▏   | 6221/10000 [42:08<13:43,  4.59it/s]
 62%|██████▏   | 6222/10000 [42:08<13:35,  4.63it/s]
 62%|██████▏   | 6223/10000 [42:08<13:13,  4.76it/s]
 62%|██████▏   | 6224/10000 [42:09<12:35,  5.00it/s]
 62%|██████▏   | 6225/10000 [42:09<12:07,  5.19it/s]
 62%|██████▏   | 6226/10000 [42:09<11:49,  5.32it/s]
 62%|██████▏   | 6227/10000 [42:09<12:04,  5.20it/s]
 62%|██████▏   | 6228/10000 [42:09<12:51,  4.89it/s]
 62%|██████▏   | 6229/10000 [42:10<13:26,  4.68it/s]
 62%|██████▏   | 6230/10000 [42:10<13:09,  4.77it/s]
                                                    
{'loss': 0.0999, 'grad_norm': 0.9503098726272583, 'learning_rate': 3.409383023633325e-05}

 62%|██████▏   | 6230/10000 [42:10<13:09,  4.77it/s]
 62%|██████▏   | 6231/10000 [42:10<13:01,  4.82it/s]
 62%|██████▏   | 6232/10000 [42:10<12:27,  5.04it/s]
 62%|██████▏   | 6233/10000 [42:10<12:00,  5.23it/s]
 62%|██████▏   | 6234/10000 [42:11<11:55,  5.26it/s]
 62%|██████▏   | 6235/10000 [42:11<12:12,  5.14it/s]
 62%|██████▏   | 6236/10000 [42:11<12:24,  5.05it/s]
 62%|██████▏   | 6237/10000 [42:11<12:28,  5.03it/s]
 62%|██████▏   | 6238/10000 [42:11<12:31,  5.00it/s]
 62%|██████▏   | 6239/10000 [42:12<12:23,  5.06it/s]
 62%|██████▏   | 6240/10000 [42:12<11:55,  5.25it/s]
                                                    
{'loss': 0.0858, 'grad_norm': 0.6576346755027771, 'learning_rate': 3.3937160402755894e-05}

 62%|██████▏   | 6240/10000 [42:12<11:55,  5.25it/s]
 62%|██████▏   | 6241/10000 [42:12<11:47,  5.31it/s]
 62%|██████▏   | 6242/10000 [42:12<11:55,  5.25it/s]
 62%|██████▏   | 6243/10000 [42:12<11:55,  5.25it/s]
 62%|██████▏   | 6244/10000 [42:13<12:13,  5.12it/s]
 62%|██████▏   | 6245/10000 [42:13<12:25,  5.04it/s]
 62%|██████▏   | 6246/10000 [42:13<12:28,  5.02it/s]
 62%|██████▏   | 6247/10000 [42:13<12:53,  4.85it/s]
 62%|██████▏   | 6248/10000 [42:13<12:22,  5.06it/s]
 62%|██████▏   | 6249/10000 [42:14<12:01,  5.20it/s]
 62%|██████▎   | 6250/10000 [42:14<11:39,  5.36it/s]Rank 0, Worker 0: Caching shard...Rank 0, Worker 1: Caching shard...Rank 0, Worker 2: Caching shard...Rank 0, Worker 3: Caching shard...



Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.19 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.38 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.40 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.61 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.64 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.68 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 6250: eval/loss = 0.1064 (36s)

                                                    
{'loss': 0.0827, 'grad_norm': 0.7527927756309509, 'learning_rate': 3.378066622981885e-05}

 62%|██████▎   | 6250/10000 [42:50<11:39,  5.36it/s]
 63%|██████▎   | 6251/10000 [42:50<11:24:47, 10.96s/it]
 63%|██████▎   | 6252/10000 [42:50<8:02:31,  7.72s/it] 
 63%|██████▎   | 6253/10000 [42:50<5:40:55,  5.46s/it]
 63%|██████▎   | 6254/10000 [42:50<4:01:53,  3.87s/it]
 63%|██████▎   | 6255/10000 [42:51<2:52:34,  2.76s/it]
 63%|██████▎   | 6256/10000 [42:51<2:04:07,  1.99s/it]
 63%|██████▎   | 6257/10000 [42:51<1:30:13,  1.45s/it]
 63%|██████▎   | 6258/10000 [42:51<1:06:32,  1.07s/it]
 63%|██████▎   | 6259/10000 [42:51<50:03,  1.25it/s]  
 63%|██████▎   | 6260/10000 [42:51<38:30,  1.62it/s]
                                                    
{'loss': 0.0738, 'grad_norm': 0.6476022005081177, 'learning_rate': 3.362434942891738e-05}

 63%|██████▎   | 6260/10000 [42:52<38:30,  1.62it/s]
 63%|██████▎   | 6261/10000 [42:52<30:40,  2.03it/s]
 63%|██████▎   | 6262/10000 [42:52<24:51,  2.51it/s]
 63%|██████▎   | 6263/10000 [42:52<20:41,  3.01it/s]
 63%|██████▎   | 6264/10000 [42:52<17:47,  3.50it/s]
 63%|██████▎   | 6265/10000 [42:52<15:43,  3.96it/s]
 63%|██████▎   | 6266/10000 [42:53<14:17,  4.35it/s]
 63%|██████▎   | 6267/10000 [42:53<13:20,  4.66it/s]
 63%|██████▎   | 6268/10000 [42:53<12:40,  4.91it/s]
 63%|██████▎   | 6269/10000 [42:53<12:14,  5.08it/s]
 63%|██████▎   | 6270/10000 [42:53<12:41,  4.90it/s]
                                                    
{'loss': 0.0727, 'grad_norm': 0.7047621607780457, 'learning_rate': 3.346821170950693e-05}

 63%|██████▎   | 6270/10000 [42:53<12:41,  4.90it/s]
 63%|██████▎   | 6271/10000 [42:54<12:57,  4.79it/s]
 63%|██████▎   | 6272/10000 [42:54<13:02,  4.77it/s]
 63%|██████▎   | 6273/10000 [42:54<12:50,  4.83it/s]
 63%|██████▎   | 6274/10000 [42:54<12:50,  4.84it/s]
 63%|██████▎   | 6275/10000 [42:54<12:52,  4.82it/s]
 63%|██████▎   | 6276/10000 [42:55<12:58,  4.78it/s]
 63%|██████▎   | 6277/10000 [42:55<13:04,  4.75it/s]
 63%|██████▎   | 6278/10000 [42:55<13:07,  4.73it/s]
 63%|██████▎   | 6279/10000 [42:55<13:21,  4.64it/s]
 63%|██████▎   | 6280/10000 [42:55<13:14,  4.68it/s]
                                                    
{'loss': 0.0688, 'grad_norm': 0.7539254426956177, 'learning_rate': 3.3312254779084585e-05}

 63%|██████▎   | 6280/10000 [42:55<13:14,  4.68it/s]
 63%|██████▎   | 6281/10000 [42:56<13:16,  4.67it/s]
 63%|██████▎   | 6282/10000 [42:56<12:40,  4.89it/s]
 63%|██████▎   | 6283/10000 [42:56<12:10,  5.09it/s]
 63%|██████▎   | 6284/10000 [42:56<11:50,  5.23it/s]
 63%|██████▎   | 6285/10000 [42:56<11:35,  5.34it/s]
 63%|██████▎   | 6286/10000 [42:57<11:28,  5.39it/s]
 63%|██████▎   | 6287/10000 [42:57<11:28,  5.39it/s]
 63%|██████▎   | 6288/10000 [42:57<11:43,  5.28it/s]
 63%|██████▎   | 6289/10000 [42:57<11:42,  5.28it/s]
 63%|██████▎   | 6290/10000 [42:57<11:42,  5.28it/s]
                                                    
{'loss': 0.075, 'grad_norm': 0.7155223488807678, 'learning_rate': 3.315648034317039e-05}

 63%|██████▎   | 6290/10000 [42:57<11:42,  5.28it/s]
 63%|██████▎   | 6291/10000 [42:58<11:53,  5.20it/s]
 63%|██████▎   | 6292/10000 [42:58<11:43,  5.27it/s]
 63%|██████▎   | 6293/10000 [42:58<11:35,  5.33it/s]
 63%|██████▎   | 6294/10000 [42:58<11:26,  5.40it/s]
 63%|██████▎   | 6295/10000 [42:58<11:18,  5.46it/s]
 63%|██████▎   | 6296/10000 [42:58<11:23,  5.42it/s]
 63%|██████▎   | 6297/10000 [42:59<11:27,  5.38it/s]
 63%|██████▎   | 6298/10000 [42:59<11:20,  5.44it/s]
 63%|██████▎   | 6299/10000 [42:59<11:19,  5.45it/s]
 63%|██████▎   | 6300/10000 [42:59<11:13,  5.49it/s]
                                                    
{'loss': 0.0865, 'grad_norm': 0.9538476467132568, 'learning_rate': 3.3000890105288564e-05}

 63%|██████▎   | 6300/10000 [42:59<11:13,  5.49it/s]
 63%|██████▎   | 6301/10000 [42:59<11:16,  5.47it/s]
 63%|██████▎   | 6302/10000 [43:00<11:07,  5.54it/s]
 63%|██████▎   | 6303/10000 [43:00<11:11,  5.51it/s]
 63%|██████▎   | 6304/10000 [43:00<11:09,  5.52it/s]
 63%|██████▎   | 6305/10000 [43:00<11:10,  5.51it/s]
 63%|██████▎   | 6306/10000 [43:00<11:18,  5.44it/s]
 63%|██████▎   | 6307/10000 [43:00<11:13,  5.49it/s]
 63%|██████▎   | 6308/10000 [43:01<11:06,  5.54it/s]
 63%|██████▎   | 6309/10000 [43:01<11:08,  5.52it/s]
 63%|██████▎   | 6310/10000 [43:01<11:08,  5.52it/s]
                                                    
{'loss': 0.0908, 'grad_norm': 0.6541961431503296, 'learning_rate': 3.284548576694908e-05}

 63%|██████▎   | 6310/10000 [43:01<11:08,  5.52it/s]
 63%|██████▎   | 6311/10000 [43:01<11:20,  5.42it/s]
 63%|██████▎   | 6312/10000 [43:01<11:44,  5.23it/s]
 63%|██████▎   | 6313/10000 [43:02<11:21,  5.41it/s]
 63%|██████▎   | 6314/10000 [43:02<11:13,  5.47it/s]
 63%|██████▎   | 6315/10000 [43:02<11:06,  5.53it/s]
 63%|██████▎   | 6316/10000 [43:02<10:56,  5.61it/s]
 63%|██████▎   | 6317/10000 [43:02<10:54,  5.63it/s]
 63%|██████▎   | 6318/10000 [43:02<10:55,  5.61it/s]
 63%|██████▎   | 6319/10000 [43:03<10:52,  5.64it/s]
 63%|██████▎   | 6320/10000 [43:03<11:07,  5.51it/s]
                                                    
{'loss': 0.0743, 'grad_norm': 0.8825085759162903, 'learning_rate': 3.2690269027628815e-05}

 63%|██████▎   | 6320/10000 [43:03<11:07,  5.51it/s]
 63%|██████▎   | 6321/10000 [43:03<11:30,  5.33it/s]
 63%|██████▎   | 6322/10000 [43:03<12:00,  5.10it/s]
 63%|██████▎   | 6323/10000 [43:03<11:46,  5.21it/s]
 63%|██████▎   | 6324/10000 [43:04<11:39,  5.25it/s]
 63%|██████▎   | 6325/10000 [43:04<11:42,  5.23it/s]
 63%|██████▎   | 6326/10000 [43:04<11:39,  5.25it/s]
 63%|██████▎   | 6327/10000 [43:04<11:40,  5.24it/s]
 63%|██████▎   | 6328/10000 [43:04<11:41,  5.23it/s]
 63%|██████▎   | 6329/10000 [43:05<11:37,  5.26it/s]
 63%|██████▎   | 6330/10000 [43:05<12:08,  5.04it/s]
                                                    
{'loss': 0.0983, 'grad_norm': 0.7695831656455994, 'learning_rate': 3.253524158475324e-05}

 63%|██████▎   | 6330/10000 [43:05<12:08,  5.04it/s]
 63%|██████▎   | 6331/10000 [43:05<12:39,  4.83it/s]
 63%|██████▎   | 6332/10000 [43:05<12:29,  4.89it/s]
 63%|██████▎   | 6333/10000 [43:05<12:12,  5.01it/s]
 63%|██████▎   | 6334/10000 [43:06<12:05,  5.05it/s]
 63%|██████▎   | 6335/10000 [43:06<12:05,  5.05it/s]
 63%|██████▎   | 6336/10000 [43:06<11:55,  5.12it/s]
 63%|██████▎   | 6337/10000 [43:06<11:42,  5.21it/s]
 63%|██████▎   | 6338/10000 [43:06<11:41,  5.22it/s]
 63%|██████▎   | 6339/10000 [43:07<11:50,  5.15it/s]
 63%|██████▎   | 6340/10000 [43:07<11:51,  5.15it/s]
                                                    
{'loss': 0.1018, 'grad_norm': 0.6998183727264404, 'learning_rate': 3.238040513367757e-05}

 63%|██████▎   | 6340/10000 [43:07<11:51,  5.15it/s]
 63%|██████▎   | 6341/10000 [43:07<12:04,  5.05it/s]
 63%|██████▎   | 6342/10000 [43:07<11:52,  5.13it/s]
 63%|██████▎   | 6343/10000 [43:07<11:35,  5.26it/s]
 63%|██████▎   | 6344/10000 [43:07<11:24,  5.34it/s]
 63%|██████▎   | 6345/10000 [43:08<11:12,  5.43it/s]
 63%|██████▎   | 6346/10000 [43:08<11:07,  5.47it/s]
 63%|██████▎   | 6347/10000 [43:08<11:48,  5.16it/s]
 63%|██████▎   | 6348/10000 [43:08<11:45,  5.18it/s]
 63%|██████▎   | 6349/10000 [43:08<11:39,  5.22it/s]
 64%|██████▎   | 6350/10000 [43:09<11:36,  5.24it/s]
                                                    
{'loss': 0.0837, 'grad_norm': 0.6837271451950073, 'learning_rate': 3.222576136766843e-05}

 64%|██████▎   | 6350/10000 [43:09<11:36,  5.24it/s]
 64%|██████▎   | 6351/10000 [43:09<11:38,  5.22it/s]
 64%|██████▎   | 6352/10000 [43:09<11:23,  5.33it/s]
 64%|██████▎   | 6353/10000 [43:09<11:23,  5.33it/s]
 64%|██████▎   | 6354/10000 [43:09<11:31,  5.27it/s]
 64%|██████▎   | 6355/10000 [43:10<11:35,  5.24it/s]
 64%|██████▎   | 6356/10000 [43:10<11:32,  5.26it/s]
 64%|██████▎   | 6357/10000 [43:10<11:29,  5.28it/s]
 64%|██████▎   | 6358/10000 [43:10<11:31,  5.27it/s]
 64%|██████▎   | 6359/10000 [43:10<11:33,  5.25it/s]
 64%|██████▎   | 6360/10000 [43:11<11:36,  5.23it/s]
                                                    
{'loss': 0.0903, 'grad_norm': 0.6960287094116211, 'learning_rate': 3.2071311977885324e-05}

 64%|██████▎   | 6360/10000 [43:11<11:36,  5.23it/s]
 64%|██████▎   | 6361/10000 [43:11<11:45,  5.16it/s]
 64%|██████▎   | 6362/10000 [43:11<11:39,  5.20it/s]
 64%|██████▎   | 6363/10000 [43:11<11:24,  5.31it/s]
 64%|██████▎   | 6364/10000 [43:11<11:16,  5.38it/s]
 64%|██████▎   | 6365/10000 [43:11<11:23,  5.32it/s]
 64%|██████▎   | 6366/10000 [43:12<11:27,  5.29it/s]
 64%|██████▎   | 6367/10000 [43:12<11:32,  5.25it/s]
 64%|██████▎   | 6368/10000 [43:12<12:05,  5.00it/s]
 64%|██████▎   | 6369/10000 [43:12<12:27,  4.86it/s]
 64%|██████▎   | 6370/10000 [43:12<12:28,  4.85it/s]
                                                    
{'loss': 0.087, 'grad_norm': 1.0823237895965576, 'learning_rate': 3.191705865336197e-05}

 64%|██████▎   | 6370/10000 [43:13<12:28,  4.85it/s]
 64%|██████▎   | 6371/10000 [43:13<12:55,  4.68it/s]
 64%|██████▎   | 6372/10000 [43:13<12:31,  4.83it/s]
 64%|██████▎   | 6373/10000 [43:13<12:10,  4.96it/s]
 64%|██████▎   | 6374/10000 [43:13<11:59,  5.04it/s]
 64%|██████▍   | 6375/10000 [43:13<11:43,  5.16it/s]
 64%|██████▍   | 6376/10000 [43:14<11:37,  5.20it/s]
 64%|██████▍   | 6377/10000 [43:14<11:37,  5.20it/s]
 64%|██████▍   | 6378/10000 [43:14<11:24,  5.29it/s]
 64%|██████▍   | 6379/10000 [43:14<11:34,  5.22it/s]
 64%|██████▍   | 6380/10000 [43:14<11:50,  5.10it/s]
                                                    
{'loss': 0.091, 'grad_norm': 0.7919114828109741, 'learning_rate': 3.1763003080988075e-05}

 64%|██████▍   | 6380/10000 [43:14<11:50,  5.10it/s]
 64%|██████▍   | 6381/10000 [43:15<12:15,  4.92it/s]
 64%|██████▍   | 6382/10000 [43:15<12:25,  4.85it/s]
 64%|██████▍   | 6383/10000 [43:15<12:24,  4.86it/s]
 64%|██████▍   | 6384/10000 [43:15<12:36,  4.78it/s]
 64%|██████▍   | 6385/10000 [43:15<12:14,  4.92it/s]
 64%|██████▍   | 6386/10000 [43:16<11:56,  5.04it/s]
 64%|██████▍   | 6387/10000 [43:16<11:53,  5.07it/s]
 64%|██████▍   | 6388/10000 [43:16<12:18,  4.89it/s]
 64%|██████▍   | 6389/10000 [43:16<12:02,  5.00it/s]
 64%|██████▍   | 6390/10000 [43:16<12:14,  4.92it/s]
                                                    
{'loss': 0.0823, 'grad_norm': 0.6094181537628174, 'learning_rate': 3.160914694549063e-05}

 64%|██████▍   | 6390/10000 [43:17<12:14,  4.92it/s]
 64%|██████▍   | 6391/10000 [43:17<13:40,  4.40it/s]
 64%|██████▍   | 6392/10000 [43:17<13:12,  4.55it/s]
 64%|██████▍   | 6393/10000 [43:17<13:06,  4.59it/s]
 64%|██████▍   | 6394/10000 [43:17<12:45,  4.71it/s]
 64%|██████▍   | 6395/10000 [43:18<12:10,  4.94it/s]
 64%|██████▍   | 6396/10000 [43:18<11:42,  5.13it/s]
 64%|██████▍   | 6397/10000 [43:18<11:17,  5.32it/s]
 64%|██████▍   | 6398/10000 [43:18<11:03,  5.43it/s]
 64%|██████▍   | 6399/10000 [43:18<11:10,  5.37it/s]
 64%|██████▍   | 6400/10000 [43:18<11:07,  5.39it/s]
                                                    
{'loss': 0.0727, 'grad_norm': 0.8500152826309204, 'learning_rate': 3.145549192941573e-05}

 64%|██████▍   | 6400/10000 [43:18<11:07,  5.39it/s]
 64%|██████▍   | 6401/10000 [43:19<11:22,  5.27it/s]
 64%|██████▍   | 6402/10000 [43:19<11:31,  5.21it/s]
 64%|██████▍   | 6403/10000 [43:19<11:39,  5.14it/s]
 64%|██████▍   | 6404/10000 [43:19<11:31,  5.20it/s]
 64%|██████▍   | 6405/10000 [43:19<11:40,  5.13it/s]
 64%|██████▍   | 6406/10000 [43:20<11:33,  5.18it/s]
 64%|██████▍   | 6407/10000 [43:20<11:18,  5.29it/s]
 64%|██████▍   | 6408/10000 [43:20<11:18,  5.29it/s]
 64%|██████▍   | 6409/10000 [43:20<11:17,  5.30it/s]
 64%|██████▍   | 6410/10000 [43:20<11:23,  5.25it/s]
                                                    
{'loss': 0.0915, 'grad_norm': 0.754837155342102, 'learning_rate': 3.130203971310999e-05}

 64%|██████▍   | 6410/10000 [43:20<11:23,  5.25it/s]
 64%|██████▍   | 6411/10000 [43:21<11:46,  5.08it/s]
 64%|██████▍   | 6412/10000 [43:21<11:42,  5.10it/s]
 64%|██████▍   | 6413/10000 [43:21<11:42,  5.10it/s]
 64%|██████▍   | 6414/10000 [43:21<11:28,  5.21it/s]
 64%|██████▍   | 6415/10000 [43:21<11:13,  5.32it/s]
 64%|██████▍   | 6416/10000 [43:22<11:09,  5.35it/s]
 64%|██████▍   | 6417/10000 [43:22<11:05,  5.38it/s]
 64%|██████▍   | 6418/10000 [43:22<11:09,  5.35it/s]
 64%|██████▍   | 6419/10000 [43:22<11:16,  5.30it/s]
 64%|██████▍   | 6420/10000 [43:22<11:07,  5.36it/s]
                                                    
{'loss': 0.0852, 'grad_norm': 0.6834582090377808, 'learning_rate': 3.114879197470225e-05}

 64%|██████▍   | 6420/10000 [43:22<11:07,  5.36it/s]
 64%|██████▍   | 6421/10000 [43:22<11:05,  5.38it/s]
 64%|██████▍   | 6422/10000 [43:23<10:57,  5.45it/s]
 64%|██████▍   | 6423/10000 [43:23<10:57,  5.44it/s]
 64%|██████▍   | 6424/10000 [43:23<11:05,  5.37it/s]
 64%|██████▍   | 6425/10000 [43:23<11:17,  5.28it/s]
 64%|██████▍   | 6426/10000 [43:23<11:35,  5.14it/s]
 64%|██████▍   | 6427/10000 [43:24<11:42,  5.09it/s]
 64%|██████▍   | 6428/10000 [43:24<11:32,  5.16it/s]
 64%|██████▍   | 6429/10000 [43:24<11:21,  5.24it/s]
 64%|██████▍   | 6430/10000 [43:24<11:07,  5.35it/s]
                                                    
{'loss': 0.0898, 'grad_norm': 0.6987155079841614, 'learning_rate': 3.0995750390085285e-05}

 64%|██████▍   | 6430/10000 [43:24<11:07,  5.35it/s]
 64%|██████▍   | 6431/10000 [43:24<11:15,  5.28it/s]
 64%|██████▍   | 6432/10000 [43:25<11:19,  5.25it/s]
 64%|██████▍   | 6433/10000 [43:25<11:47,  5.04it/s]
 64%|██████▍   | 6434/10000 [43:25<12:17,  4.83it/s]
 64%|██████▍   | 6435/10000 [43:25<12:41,  4.68it/s]
 64%|██████▍   | 6436/10000 [43:25<13:09,  4.51it/s]
 64%|██████▍   | 6437/10000 [43:26<13:12,  4.50it/s]
 64%|██████▍   | 6438/10000 [43:26<12:27,  4.77it/s]
 64%|██████▍   | 6439/10000 [43:26<11:59,  4.95it/s]
 64%|██████▍   | 6440/10000 [43:26<11:49,  5.02it/s]
                                                    
{'loss': 0.0779, 'grad_norm': 0.5847340226173401, 'learning_rate': 3.084291663289728e-05}

 64%|██████▍   | 6440/10000 [43:26<11:49,  5.02it/s]
 64%|██████▍   | 6441/10000 [43:26<11:48,  5.02it/s]
 64%|██████▍   | 6442/10000 [43:27<12:01,  4.93it/s]
 64%|██████▍   | 6443/10000 [43:27<12:27,  4.76it/s]
 64%|██████▍   | 6444/10000 [43:27<12:30,  4.74it/s]
 64%|██████▍   | 6445/10000 [43:27<12:27,  4.76it/s]
 64%|██████▍   | 6446/10000 [43:28<12:24,  4.77it/s]
 64%|██████▍   | 6447/10000 [43:28<11:51,  4.99it/s]
 64%|██████▍   | 6448/10000 [43:28<11:32,  5.13it/s]
 64%|██████▍   | 6449/10000 [43:28<11:31,  5.13it/s]
 64%|██████▍   | 6450/10000 [43:28<11:31,  5.14it/s]
                                                    
{'loss': 0.0728, 'grad_norm': 0.6000155210494995, 'learning_rate': 3.069029237450375e-05}

 64%|██████▍   | 6450/10000 [43:28<11:31,  5.14it/s]
 65%|██████▍   | 6451/10000 [43:29<12:46,  4.63it/s]
 65%|██████▍   | 6452/10000 [43:29<12:54,  4.58it/s]
 65%|██████▍   | 6453/10000 [43:29<13:03,  4.53it/s]
 65%|██████▍   | 6454/10000 [43:29<12:44,  4.64it/s]
 65%|██████▍   | 6455/10000 [43:29<12:29,  4.73it/s]
 65%|██████▍   | 6456/10000 [43:30<12:16,  4.81it/s]
 65%|██████▍   | 6457/10000 [43:30<11:52,  4.97it/s]
 65%|██████▍   | 6458/10000 [43:30<11:41,  5.05it/s]
 65%|██████▍   | 6459/10000 [43:30<11:42,  5.04it/s]
 65%|██████▍   | 6460/10000 [43:30<11:44,  5.03it/s]
                                                    
{'loss': 0.0967, 'grad_norm': 0.6943097710609436, 'learning_rate': 3.053787928397911e-05}

 65%|██████▍   | 6460/10000 [43:30<11:44,  5.03it/s]
 65%|██████▍   | 6461/10000 [43:31<12:34,  4.69it/s]
 65%|██████▍   | 6462/10000 [43:31<12:37,  4.67it/s]
 65%|██████▍   | 6463/10000 [43:31<12:35,  4.68it/s]
 65%|██████▍   | 6464/10000 [43:31<12:29,  4.72it/s]
 65%|██████▍   | 6465/10000 [43:31<12:35,  4.68it/s]
 65%|██████▍   | 6466/10000 [43:32<12:28,  4.72it/s]
 65%|██████▍   | 6467/10000 [43:32<11:52,  4.96it/s]
 65%|██████▍   | 6468/10000 [43:32<11:56,  4.93it/s]
 65%|██████▍   | 6469/10000 [43:32<11:55,  4.94it/s]
 65%|██████▍   | 6470/10000 [43:32<12:09,  4.84it/s]
                                                    
{'loss': 0.0845, 'grad_norm': 0.6858120560646057, 'learning_rate': 3.0385679028088526e-05}

 65%|██████▍   | 6470/10000 [43:33<12:09,  4.84it/s]
 65%|██████▍   | 6471/10000 [43:33<12:53,  4.57it/s]
 65%|██████▍   | 6472/10000 [43:33<13:05,  4.49it/s]
 65%|██████▍   | 6473/10000 [43:33<12:54,  4.55it/s]
 65%|██████▍   | 6474/10000 [43:33<12:29,  4.71it/s]
 65%|██████▍   | 6475/10000 [43:34<11:50,  4.96it/s]
 65%|██████▍   | 6476/10000 [43:34<11:29,  5.11it/s]
 65%|██████▍   | 6477/10000 [43:34<11:34,  5.07it/s]
 65%|██████▍   | 6478/10000 [43:34<11:43,  5.01it/s]
 65%|██████▍   | 6479/10000 [43:34<12:11,  4.81it/s]
 65%|██████▍   | 6480/10000 [43:35<12:33,  4.67it/s]
                                                    
{'loss': 0.0846, 'grad_norm': 0.7228976488113403, 'learning_rate': 3.023369327126959e-05}

 65%|██████▍   | 6480/10000 [43:35<12:33,  4.67it/s]
 65%|██████▍   | 6481/10000 [43:35<12:45,  4.59it/s]
 65%|██████▍   | 6482/10000 [43:35<12:24,  4.72it/s]
 65%|██████▍   | 6483/10000 [43:35<11:46,  4.98it/s]
 65%|██████▍   | 6484/10000 [43:35<11:20,  5.16it/s]
 65%|██████▍   | 6485/10000 [43:36<11:09,  5.25it/s]
 65%|██████▍   | 6486/10000 [43:36<11:23,  5.14it/s]
 65%|██████▍   | 6487/10000 [43:36<11:44,  4.99it/s]
 65%|██████▍   | 6488/10000 [43:36<12:20,  4.74it/s]
 65%|██████▍   | 6489/10000 [43:36<12:44,  4.59it/s]
 65%|██████▍   | 6490/10000 [43:37<12:57,  4.52it/s]
                                                    
{'loss': 0.0843, 'grad_norm': 0.6238749027252197, 'learning_rate': 3.0081923675614198e-05}

 65%|██████▍   | 6490/10000 [43:37<12:57,  4.52it/s]
 65%|██████▍   | 6491/10000 [43:37<12:37,  4.63it/s]
 65%|██████▍   | 6492/10000 [43:37<12:00,  4.87it/s]
 65%|██████▍   | 6493/10000 [43:37<11:38,  5.02it/s]
 65%|██████▍   | 6494/10000 [43:37<11:43,  4.98it/s]
 65%|██████▍   | 6495/10000 [43:38<11:51,  4.93it/s]
 65%|██████▍   | 6496/10000 [43:38<11:57,  4.89it/s]
 65%|██████▍   | 6497/10000 [43:38<12:21,  4.73it/s]
 65%|██████▍   | 6498/10000 [43:38<12:07,  4.81it/s]
 65%|██████▍   | 6499/10000 [43:38<11:51,  4.92it/s]
 65%|██████▌   | 6500/10000 [43:39<11:29,  5.08it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.23 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.79 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 18.93 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.03 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.36 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.77 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 6500: eval/loss = 0.0951 (37s)

                                                    
{'loss': 0.073, 'grad_norm': 0.6193327307701111, 'learning_rate': 2.993037190085034e-05}

 65%|██████▌   | 6500/10000 [44:16<11:29,  5.08it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-6500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-6500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-6500/wandb_config.json

 65%|██████▌   | 6501/10000 [44:43<18:53:32, 19.44s/it]
 65%|██████▌   | 6502/10000 [44:43<13:16:50, 13.67s/it]
 65%|██████▌   | 6503/10000 [44:43<9:21:02,  9.63s/it] 
 65%|██████▌   | 6504/10000 [44:44<6:35:51,  6.79s/it]
 65%|██████▌   | 6505/10000 [44:44<4:40:14,  4.81s/it]
 65%|██████▌   | 6506/10000 [44:44<3:19:23,  3.42s/it]
 65%|██████▌   | 6507/10000 [44:44<2:22:49,  2.45s/it]
 65%|██████▌   | 6508/10000 [44:44<1:43:15,  1.77s/it]
 65%|██████▌   | 6509/10000 [44:45<1:15:33,  1.30s/it]
 65%|██████▌   | 6510/10000 [44:45<56:12,  1.03it/s]  
                                                    
{'loss': 0.0726, 'grad_norm': 0.6352846622467041, 'learning_rate': 2.977903960432392e-05}

 65%|██████▌   | 6510/10000 [44:45<56:12,  1.03it/s]
 65%|██████▌   | 6511/10000 [44:45<42:47,  1.36it/s]
 65%|██████▌   | 6512/10000 [44:45<33:07,  1.75it/s]
 65%|██████▌   | 6513/10000 [44:45<26:20,  2.21it/s]
 65%|██████▌   | 6514/10000 [44:45<21:34,  2.69it/s]
 65%|██████▌   | 6515/10000 [44:46<18:16,  3.18it/s]
 65%|██████▌   | 6516/10000 [44:46<16:18,  3.56it/s]
 65%|██████▌   | 6517/10000 [44:46<14:32,  3.99it/s]
 65%|██████▌   | 6518/10000 [44:46<13:14,  4.38it/s]
 65%|██████▌   | 6519/10000 [44:46<12:17,  4.72it/s]
 65%|██████▌   | 6520/10000 [44:47<11:45,  4.93it/s]
                                                    
{'loss': 0.0667, 'grad_norm': 0.6984387040138245, 'learning_rate': 2.9627928440980722e-05}

 65%|██████▌   | 6520/10000 [44:47<11:45,  4.93it/s]
 65%|██████▌   | 6521/10000 [44:47<11:24,  5.08it/s]
 65%|██████▌   | 6522/10000 [44:47<11:11,  5.18it/s]
 65%|██████▌   | 6523/10000 [44:47<11:05,  5.22it/s]
 65%|██████▌   | 6524/10000 [44:47<11:04,  5.23it/s]
 65%|██████▌   | 6525/10000 [44:47<11:03,  5.24it/s]
 65%|██████▌   | 6526/10000 [44:48<11:02,  5.24it/s]
 65%|██████▌   | 6527/10000 [44:48<10:47,  5.37it/s]
 65%|██████▌   | 6528/10000 [44:48<10:44,  5.39it/s]
 65%|██████▌   | 6529/10000 [44:48<10:40,  5.42it/s]
 65%|██████▌   | 6530/10000 [44:48<10:43,  5.39it/s]
                                                    
{'loss': 0.0751, 'grad_norm': 0.583076000213623, 'learning_rate': 2.9477040063348183e-05}

 65%|██████▌   | 6530/10000 [44:48<10:43,  5.39it/s]
 65%|██████▌   | 6531/10000 [44:49<10:57,  5.28it/s]
 65%|██████▌   | 6532/10000 [44:49<11:00,  5.25it/s]
 65%|██████▌   | 6533/10000 [44:49<10:59,  5.26it/s]
 65%|██████▌   | 6534/10000 [44:49<11:02,  5.23it/s]
 65%|██████▌   | 6535/10000 [44:49<11:24,  5.07it/s]
 65%|██████▌   | 6536/10000 [44:50<12:07,  4.76it/s]
 65%|██████▌   | 6537/10000 [44:50<12:15,  4.71it/s]
 65%|██████▌   | 6538/10000 [44:50<12:25,  4.64it/s]
 65%|██████▌   | 6539/10000 [44:50<12:47,  4.51it/s]
 65%|██████▌   | 6540/10000 [44:50<12:14,  4.71it/s]
                                                    
{'loss': 0.0764, 'grad_norm': 0.7024626135826111, 'learning_rate': 2.9326376121517456e-05}

 65%|██████▌   | 6540/10000 [44:51<12:14,  4.71it/s]
 65%|██████▌   | 6541/10000 [44:51<11:49,  4.87it/s]
 65%|██████▌   | 6542/10000 [44:51<11:23,  5.06it/s]
 65%|██████▌   | 6543/10000 [44:51<11:21,  5.07it/s]
 65%|██████▌   | 6544/10000 [44:51<11:18,  5.10it/s]
 65%|██████▌   | 6545/10000 [44:51<11:43,  4.91it/s]
 65%|██████▌   | 6546/10000 [44:52<12:07,  4.75it/s]
 65%|██████▌   | 6547/10000 [44:52<12:06,  4.75it/s]
 65%|██████▌   | 6548/10000 [44:52<12:03,  4.77it/s]
 65%|██████▌   | 6549/10000 [44:52<11:57,  4.81it/s]
 66%|██████▌   | 6550/10000 [44:52<11:29,  5.00it/s]
                                                    
{'loss': 0.081, 'grad_norm': 0.8099085688591003, 'learning_rate': 2.9175938263125236e-05}

 66%|██████▌   | 6550/10000 [44:53<11:29,  5.00it/s]
 66%|██████▌   | 6551/10000 [44:53<11:38,  4.94it/s]
 66%|██████▌   | 6552/10000 [44:53<11:31,  4.99it/s]
 66%|██████▌   | 6553/10000 [44:53<11:27,  5.02it/s]
 66%|██████▌   | 6554/10000 [44:53<11:27,  5.01it/s]
 66%|██████▌   | 6555/10000 [44:54<11:43,  4.90it/s]
 66%|██████▌   | 6556/10000 [44:54<12:13,  4.70it/s]
 66%|██████▌   | 6557/10000 [44:54<12:40,  4.53it/s]
 66%|██████▌   | 6558/10000 [44:54<12:37,  4.55it/s]
 66%|██████▌   | 6559/10000 [44:54<12:29,  4.59it/s]
 66%|██████▌   | 6560/10000 [44:55<12:01,  4.77it/s]
                                                    
{'loss': 0.1018, 'grad_norm': 0.7837961316108704, 'learning_rate': 2.9025728133335873e-05}

 66%|██████▌   | 6560/10000 [44:55<12:01,  4.77it/s]
 66%|██████▌   | 6561/10000 [44:55<11:57,  4.79it/s]
 66%|██████▌   | 6562/10000 [44:55<12:00,  4.77it/s]
 66%|██████▌   | 6563/10000 [44:55<12:08,  4.72it/s]
 66%|██████▌   | 6564/10000 [44:55<12:11,  4.69it/s]
 66%|██████▌   | 6565/10000 [44:56<12:18,  4.65it/s]
 66%|██████▌   | 6566/10000 [44:56<12:17,  4.66it/s]
 66%|██████▌   | 6567/10000 [44:56<12:22,  4.62it/s]
 66%|██████▌   | 6568/10000 [44:56<12:17,  4.65it/s]
 66%|██████▌   | 6569/10000 [44:57<11:58,  4.78it/s]
 66%|██████▌   | 6570/10000 [44:57<11:45,  4.86it/s]
                                                    
{'loss': 0.0977, 'grad_norm': 0.7633925080299377, 'learning_rate': 2.8875747374823288e-05}

 66%|██████▌   | 6570/10000 [44:57<11:45,  4.86it/s]
 66%|██████▌   | 6571/10000 [44:57<11:42,  4.88it/s]
 66%|██████▌   | 6572/10000 [44:57<11:38,  4.91it/s]
 66%|██████▌   | 6573/10000 [44:57<11:57,  4.78it/s]
 66%|██████▌   | 6574/10000 [44:58<12:44,  4.48it/s]
 66%|██████▌   | 6575/10000 [44:58<12:40,  4.51it/s]
 66%|██████▌   | 6576/10000 [44:58<12:40,  4.50it/s]
 66%|██████▌   | 6577/10000 [44:58<12:00,  4.75it/s]
 66%|██████▌   | 6578/10000 [44:58<11:33,  4.94it/s]
 66%|██████▌   | 6579/10000 [44:59<11:21,  5.02it/s]
 66%|██████▌   | 6580/10000 [44:59<11:26,  4.98it/s]
                                                    
{'loss': 0.0844, 'grad_norm': 0.6811825037002563, 'learning_rate': 2.872599762775298e-05}

 66%|██████▌   | 6580/10000 [44:59<11:26,  4.98it/s]
 66%|██████▌   | 6581/10000 [44:59<12:01,  4.74it/s]
 66%|██████▌   | 6582/10000 [44:59<12:13,  4.66it/s]
 66%|██████▌   | 6583/10000 [44:59<12:18,  4.63it/s]
 66%|██████▌   | 6584/10000 [45:00<12:32,  4.54it/s]
 66%|██████▌   | 6585/10000 [45:00<12:13,  4.66it/s]
 66%|██████▌   | 6586/10000 [45:00<11:36,  4.90it/s]
 66%|██████▌   | 6587/10000 [45:00<11:47,  4.83it/s]
 66%|██████▌   | 6588/10000 [45:01<12:16,  4.63it/s]
 66%|██████▌   | 6589/10000 [45:01<12:34,  4.52it/s]
 66%|██████▌   | 6590/10000 [45:01<15:50,  3.59it/s]
                                                    
{'loss': 0.0697, 'grad_norm': 0.568018913269043, 'learning_rate': 2.857648052976425e-05}

 66%|██████▌   | 6590/10000 [45:01<15:50,  3.59it/s]
 66%|██████▌   | 6591/10000 [45:01<14:43,  3.86it/s]
 66%|██████▌   | 6592/10000 [45:02<13:30,  4.21it/s]
 66%|██████▌   | 6593/10000 [45:02<12:34,  4.51it/s]
 66%|██████▌   | 6594/10000 [45:02<12:15,  4.63it/s]
 66%|██████▌   | 6595/10000 [45:02<12:17,  4.62it/s]
 66%|██████▌   | 6596/10000 [45:02<13:04,  4.34it/s]
 66%|██████▌   | 6597/10000 [45:03<13:47,  4.11it/s]
 66%|██████▌   | 6598/10000 [45:03<14:04,  4.03it/s]
 66%|██████▌   | 6599/10000 [45:03<13:50,  4.10it/s]
 66%|██████▌   | 6600/10000 [45:03<13:36,  4.16it/s]
                                                    
{'loss': 0.1085, 'grad_norm': 0.8331983089447021, 'learning_rate': 2.8427197715952047e-05}

 66%|██████▌   | 6600/10000 [45:03<13:36,  4.16it/s]
 66%|██████▌   | 6601/10000 [45:04<13:24,  4.23it/s]
 66%|██████▌   | 6602/10000 [45:04<13:11,  4.29it/s]
 66%|██████▌   | 6603/10000 [45:04<13:00,  4.35it/s]
 66%|██████▌   | 6604/10000 [45:04<13:19,  4.25it/s]
 66%|██████▌   | 6605/10000 [45:05<13:24,  4.22it/s]
 66%|██████▌   | 6606/10000 [45:05<14:13,  3.98it/s]
 66%|██████▌   | 6607/10000 [45:05<13:31,  4.18it/s]
 66%|██████▌   | 6608/10000 [45:05<13:09,  4.30it/s]
 66%|██████▌   | 6609/10000 [45:06<12:39,  4.46it/s]
 66%|██████▌   | 6610/10000 [45:06<12:11,  4.63it/s]
                                                    
{'loss': 0.0794, 'grad_norm': 0.653430700302124, 'learning_rate': 2.8278150818849393e-05}

 66%|██████▌   | 6610/10000 [45:06<12:11,  4.63it/s]
 66%|██████▌   | 6611/10000 [45:06<12:08,  4.65it/s]
 66%|██████▌   | 6612/10000 [45:06<12:16,  4.60it/s]
 66%|██████▌   | 6613/10000 [45:06<12:30,  4.51it/s]
 66%|██████▌   | 6614/10000 [45:07<12:57,  4.36it/s]
 66%|██████▌   | 6615/10000 [45:07<13:13,  4.26it/s]
 66%|██████▌   | 6616/10000 [45:07<13:24,  4.21it/s]
 66%|██████▌   | 6617/10000 [45:07<13:04,  4.31it/s]
 66%|██████▌   | 6618/10000 [45:08<12:53,  4.37it/s]
 66%|██████▌   | 6619/10000 [45:08<12:58,  4.34it/s]
 66%|██████▌   | 6620/10000 [45:08<12:25,  4.53it/s]
                                                    
{'loss': 0.1057, 'grad_norm': 0.6260983347892761, 'learning_rate': 2.812934146840922e-05}

 66%|██████▌   | 6620/10000 [45:08<12:25,  4.53it/s]
 66%|██████▌   | 6621/10000 [45:08<12:14,  4.60it/s]
 66%|██████▌   | 6622/10000 [45:08<12:12,  4.61it/s]
 66%|██████▌   | 6623/10000 [45:09<12:30,  4.50it/s]
 66%|██████▌   | 6624/10000 [45:09<12:28,  4.51it/s]
 66%|██████▋   | 6625/10000 [45:09<12:26,  4.52it/s]
 66%|██████▋   | 6626/10000 [45:09<12:19,  4.57it/s]
 66%|██████▋   | 6627/10000 [45:10<12:12,  4.60it/s]
 66%|██████▋   | 6628/10000 [45:10<11:59,  4.69it/s]
 66%|██████▋   | 6629/10000 [45:10<11:50,  4.74it/s]
 66%|██████▋   | 6630/10000 [45:10<11:37,  4.83it/s]
                                                    
{'loss': 0.0846, 'grad_norm': 0.793511688709259, 'learning_rate': 2.7980771291986764e-05}

 66%|██████▋   | 6630/10000 [45:10<11:37,  4.83it/s]
 66%|██████▋   | 6631/10000 [45:10<11:37,  4.83it/s]
 66%|██████▋   | 6632/10000 [45:11<11:43,  4.79it/s]
 66%|██████▋   | 6633/10000 [45:11<11:42,  4.79it/s]
 66%|██████▋   | 6634/10000 [45:11<11:44,  4.78it/s]
 66%|██████▋   | 6635/10000 [45:11<11:52,  4.73it/s]
 66%|██████▋   | 6636/10000 [45:11<11:49,  4.74it/s]
 66%|██████▋   | 6637/10000 [45:12<12:05,  4.63it/s]
 66%|██████▋   | 6638/10000 [45:12<12:06,  4.63it/s]
 66%|██████▋   | 6639/10000 [45:12<11:52,  4.72it/s]
 66%|██████▋   | 6640/10000 [45:12<11:28,  4.88it/s]
                                                    
{'loss': 0.0749, 'grad_norm': 0.6311041116714478, 'learning_rate': 2.783244191432167e-05}

 66%|██████▋   | 6640/10000 [45:12<11:28,  4.88it/s]
 66%|██████▋   | 6641/10000 [45:12<11:30,  4.87it/s]
 66%|██████▋   | 6642/10000 [45:13<11:27,  4.88it/s]
 66%|██████▋   | 6643/10000 [45:13<12:13,  4.58it/s]
 66%|██████▋   | 6644/10000 [45:13<12:13,  4.58it/s]
 66%|██████▋   | 6645/10000 [45:13<12:05,  4.62it/s]
 66%|██████▋   | 6646/10000 [45:14<11:56,  4.68it/s]
 66%|██████▋   | 6647/10000 [45:14<11:44,  4.76it/s]
 66%|██████▋   | 6648/10000 [45:14<11:33,  4.83it/s]
 66%|██████▋   | 6649/10000 [45:14<11:31,  4.85it/s]
 66%|██████▋   | 6650/10000 [45:14<11:29,  4.86it/s]
                                                    
{'loss': 0.0773, 'grad_norm': 0.7379437685012817, 'learning_rate': 2.768435495752022e-05}

 66%|██████▋   | 6650/10000 [45:14<11:29,  4.86it/s]
 67%|██████▋   | 6651/10000 [45:15<11:39,  4.79it/s]
 67%|██████▋   | 6652/10000 [45:15<11:35,  4.81it/s]
 67%|██████▋   | 6653/10000 [45:15<11:36,  4.80it/s]
 67%|██████▋   | 6654/10000 [45:15<11:41,  4.77it/s]
 67%|██████▋   | 6655/10000 [45:15<11:35,  4.81it/s]
 67%|██████▋   | 6656/10000 [45:16<11:24,  4.88it/s]
 67%|██████▋   | 6657/10000 [45:16<11:29,  4.85it/s]
 67%|██████▋   | 6658/10000 [45:16<11:27,  4.86it/s]
 67%|██████▋   | 6659/10000 [45:16<11:28,  4.85it/s]
 67%|██████▋   | 6660/10000 [45:16<11:26,  4.87it/s]
                                                    
{'loss': 0.0764, 'grad_norm': 0.6751238703727722, 'learning_rate': 2.753651204103771e-05}

 67%|██████▋   | 6660/10000 [45:16<11:26,  4.87it/s]
 67%|██████▋   | 6661/10000 [45:17<11:32,  4.82it/s]
 67%|██████▋   | 6662/10000 [45:17<11:41,  4.76it/s]
 67%|██████▋   | 6663/10000 [45:17<11:26,  4.86it/s]
 67%|██████▋   | 6664/10000 [45:17<11:30,  4.83it/s]
 67%|██████▋   | 6665/10000 [45:17<11:11,  4.96it/s]
 67%|██████▋   | 6666/10000 [45:18<11:04,  5.02it/s]
 67%|██████▋   | 6667/10000 [45:18<11:05,  5.01it/s]
 67%|██████▋   | 6668/10000 [45:18<11:06,  5.00it/s]
 67%|██████▋   | 6669/10000 [45:18<11:08,  4.98it/s]
 67%|██████▋   | 6670/10000 [45:18<11:11,  4.96it/s]
                                                    
{'loss': 0.0834, 'grad_norm': 0.6878944039344788, 'learning_rate': 2.7388914781660523e-05}

 67%|██████▋   | 6670/10000 [45:18<11:11,  4.96it/s]
 67%|██████▋   | 6671/10000 [45:19<11:25,  4.85it/s]
 67%|██████▋   | 6672/10000 [45:19<11:25,  4.86it/s]
 67%|██████▋   | 6673/10000 [45:19<11:13,  4.94it/s]
 67%|██████▋   | 6674/10000 [45:19<11:04,  5.01it/s]
 67%|██████▋   | 6675/10000 [45:19<11:06,  4.99it/s]
 67%|██████▋   | 6676/10000 [45:20<11:05,  4.99it/s]
 67%|██████▋   | 6677/10000 [45:20<11:10,  4.95it/s]
 67%|██████▋   | 6678/10000 [45:20<11:11,  4.95it/s]
 67%|██████▋   | 6679/10000 [45:20<11:23,  4.86it/s]
 67%|██████▋   | 6680/10000 [45:21<11:42,  4.73it/s]
                                                    
{'loss': 0.0672, 'grad_norm': 0.6290616393089294, 'learning_rate': 2.7241564793488693e-05}

 67%|██████▋   | 6680/10000 [45:21<11:42,  4.73it/s]
 67%|██████▋   | 6681/10000 [45:21<11:29,  4.81it/s]
 67%|██████▋   | 6682/10000 [45:21<11:00,  5.02it/s]
 67%|██████▋   | 6683/10000 [45:21<10:55,  5.06it/s]
 67%|██████▋   | 6684/10000 [45:21<10:53,  5.07it/s]
 67%|██████▋   | 6685/10000 [45:21<10:55,  5.06it/s]
 67%|██████▋   | 6686/10000 [45:22<10:56,  5.05it/s]
 67%|██████▋   | 6687/10000 [45:22<10:54,  5.06it/s]
 67%|██████▋   | 6688/10000 [45:22<11:10,  4.94it/s]
 67%|██████▋   | 6689/10000 [45:22<11:20,  4.87it/s]
 67%|██████▋   | 6690/10000 [45:22<11:13,  4.92it/s]
                                                    
{'loss': 0.0786, 'grad_norm': 0.6177688241004944, 'learning_rate': 2.7094463687918037e-05}

 67%|██████▋   | 6690/10000 [45:23<11:13,  4.92it/s]
 67%|██████▋   | 6691/10000 [45:23<11:05,  4.97it/s]
 67%|██████▋   | 6692/10000 [45:23<10:51,  5.07it/s]
 67%|██████▋   | 6693/10000 [45:23<10:42,  5.15it/s]
 67%|██████▋   | 6694/10000 [45:23<10:38,  5.18it/s]
 67%|██████▋   | 6695/10000 [45:23<10:47,  5.11it/s]
 67%|██████▋   | 6696/10000 [45:24<10:37,  5.18it/s]
 67%|██████▋   | 6697/10000 [45:24<10:28,  5.25it/s]
 67%|██████▋   | 6698/10000 [45:24<10:23,  5.29it/s]
 67%|██████▋   | 6699/10000 [45:24<10:25,  5.28it/s]
 67%|██████▋   | 6700/10000 [45:24<10:24,  5.28it/s]
                                                    
{'loss': 0.0693, 'grad_norm': 0.6794228553771973, 'learning_rate': 2.694761307362268e-05}

 67%|██████▋   | 6700/10000 [45:24<10:24,  5.28it/s]
 67%|██████▋   | 6701/10000 [45:25<10:35,  5.19it/s]
 67%|██████▋   | 6702/10000 [45:25<10:20,  5.32it/s]
 67%|██████▋   | 6703/10000 [45:25<10:12,  5.38it/s]
 67%|██████▋   | 6704/10000 [45:25<10:03,  5.46it/s]
 67%|██████▋   | 6705/10000 [45:25<10:03,  5.46it/s]
 67%|██████▋   | 6706/10000 [45:25<10:06,  5.43it/s]
 67%|██████▋   | 6707/10000 [45:26<09:56,  5.52it/s]
 67%|██████▋   | 6708/10000 [45:26<09:59,  5.49it/s]
 67%|██████▋   | 6709/10000 [45:26<10:04,  5.44it/s]
 67%|██████▋   | 6710/10000 [45:26<10:06,  5.42it/s]
                                                    
{'loss': 0.0715, 'grad_norm': 0.7220656275749207, 'learning_rate': 2.6801014556537467e-05}

 67%|██████▋   | 6710/10000 [45:26<10:06,  5.42it/s]
 67%|██████▋   | 6711/10000 [45:26<10:18,  5.32it/s]
 67%|██████▋   | 6712/10000 [45:27<10:29,  5.22it/s]
 67%|██████▋   | 6713/10000 [45:27<10:31,  5.21it/s]
 67%|██████▋   | 6714/10000 [45:27<10:22,  5.28it/s]
 67%|██████▋   | 6715/10000 [45:27<10:26,  5.24it/s]
 67%|██████▋   | 6716/10000 [45:27<10:06,  5.41it/s]
 67%|██████▋   | 6717/10000 [45:28<10:00,  5.47it/s]
 67%|██████▋   | 6718/10000 [45:28<09:58,  5.48it/s]
 67%|██████▋   | 6719/10000 [45:28<09:52,  5.54it/s]
 67%|██████▋   | 6720/10000 [45:28<10:03,  5.43it/s]
                                                    
{'loss': 0.0714, 'grad_norm': 0.6824714541435242, 'learning_rate': 2.6654669739840243e-05}

 67%|██████▋   | 6720/10000 [45:28<10:03,  5.43it/s]
 67%|██████▋   | 6721/10000 [45:28<10:32,  5.18it/s]
 67%|██████▋   | 6722/10000 [45:29<10:40,  5.12it/s]
 67%|██████▋   | 6723/10000 [45:29<10:42,  5.10it/s]
 67%|██████▋   | 6724/10000 [45:29<10:35,  5.15it/s]
 67%|██████▋   | 6725/10000 [45:29<10:20,  5.28it/s]
 67%|██████▋   | 6726/10000 [45:29<10:17,  5.30it/s]
 67%|██████▋   | 6727/10000 [45:29<10:09,  5.37it/s]
 67%|██████▋   | 6728/10000 [45:30<10:14,  5.32it/s]
 67%|██████▋   | 6729/10000 [45:30<10:26,  5.22it/s]
 67%|██████▋   | 6730/10000 [45:30<10:35,  5.15it/s]
                                                    
{'loss': 0.0826, 'grad_norm': 0.8029189705848694, 'learning_rate': 2.650858022393451e-05}

 67%|██████▋   | 6730/10000 [45:30<10:35,  5.15it/s]
 67%|██████▋   | 6731/10000 [45:30<10:40,  5.11it/s]
 67%|██████▋   | 6732/10000 [45:30<10:24,  5.23it/s]
 67%|██████▋   | 6733/10000 [45:31<10:09,  5.36it/s]
 67%|██████▋   | 6734/10000 [45:31<10:07,  5.38it/s]
 67%|██████▋   | 6735/10000 [45:31<10:19,  5.27it/s]
 67%|██████▋   | 6736/10000 [45:31<10:29,  5.19it/s]
 67%|██████▋   | 6737/10000 [45:31<10:35,  5.13it/s]
 67%|██████▋   | 6738/10000 [45:32<10:37,  5.11it/s]
 67%|██████▋   | 6739/10000 [45:32<10:26,  5.20it/s]
 67%|██████▋   | 6740/10000 [45:32<10:13,  5.31it/s]
                                                    
{'loss': 0.078, 'grad_norm': 0.7607128620147705, 'learning_rate': 2.6362747606431747e-05}

 67%|██████▋   | 6740/10000 [45:32<10:13,  5.31it/s]
 67%|██████▋   | 6741/10000 [45:32<10:59,  4.94it/s]
 67%|██████▋   | 6742/10000 [45:32<10:58,  4.95it/s]
 67%|██████▋   | 6743/10000 [45:33<11:02,  4.91it/s]
 67%|██████▋   | 6744/10000 [45:33<11:14,  4.83it/s]
 67%|██████▋   | 6745/10000 [45:33<11:11,  4.85it/s]Rank 0, Worker 1: Wait for shard 38 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 67%|██████▋   | 6746/10000 [45:33<11:12,  4.84it/s]
 67%|██████▋   | 6747/10000 [45:33<11:03,  4.91it/s]
 67%|██████▋   | 6748/10000 [45:34<10:53,  4.98it/s]
 67%|██████▋   | 6749/10000 [45:34<10:44,  5.04it/s]
 68%|██████▊   | 6750/10000 [45:34<10:46,  5.03it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 3: Caching shard...

Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.64 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.83 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 16.84 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 16.97 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.45 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.54 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 6750: eval/loss = 0.0985 (35s)

                                                    
{'loss': 0.0816, 'grad_norm': 0.6437284350395203, 'learning_rate': 2.6217173482134172e-05}

 68%|██████▊   | 6750/10000 [46:09<10:46,  5.03it/s]
 68%|██████▊   | 6751/10000 [46:09<9:37:18, 10.66s/it]
 68%|██████▊   | 6752/10000 [46:09<6:46:53,  7.52s/it]
 68%|██████▊   | 6753/10000 [46:09<4:47:32,  5.31s/it]
 68%|██████▊   | 6754/10000 [46:10<3:24:06,  3.77s/it]
 68%|██████▊   | 6755/10000 [46:10<2:25:43,  2.69s/it]
 68%|██████▊   | 6756/10000 [46:10<1:45:01,  1.94s/it]
 68%|██████▊   | 6757/10000 [46:10<1:17:01,  1.43s/it]
 68%|██████▊   | 6758/10000 [46:10<57:21,  1.06s/it]  
 68%|██████▊   | 6759/10000 [46:11<43:12,  1.25it/s]
 68%|██████▊   | 6760/10000 [46:11<33:35,  1.61it/s]
                                                    
{'loss': 0.0645, 'grad_norm': 0.641262412071228, 'learning_rate': 2.6071859443017044e-05}

 68%|██████▊   | 6760/10000 [46:11<33:35,  1.61it/s]
 68%|██████▊   | 6761/10000 [46:11<27:00,  2.00it/s]
 68%|██████▊   | 6762/10000 [46:11<22:19,  2.42it/s]
 68%|██████▊   | 6763/10000 [46:11<18:57,  2.85it/s]
 68%|██████▊   | 6764/10000 [46:12<16:37,  3.24it/s]
 68%|██████▊   | 6765/10000 [46:12<14:49,  3.64it/s]
 68%|██████▊   | 6766/10000 [46:12<13:39,  3.95it/s]
 68%|██████▊   | 6767/10000 [46:12<12:50,  4.19it/s]
 68%|██████▊   | 6768/10000 [46:12<12:28,  4.32it/s]
 68%|██████▊   | 6769/10000 [46:13<12:21,  4.35it/s]
 68%|██████▊   | 6770/10000 [46:13<12:04,  4.46it/s]
                                                    
{'loss': 0.0806, 'grad_norm': 0.8665381669998169, 'learning_rate': 2.5926807078211414e-05}

 68%|██████▊   | 6770/10000 [46:13<12:04,  4.46it/s]
 68%|██████▊   | 6771/10000 [46:13<11:59,  4.49it/s]
 68%|██████▊   | 6772/10000 [46:13<11:47,  4.56it/s]
 68%|██████▊   | 6773/10000 [46:13<11:04,  4.85it/s]
 68%|██████▊   | 6774/10000 [46:14<10:39,  5.05it/s]
 68%|██████▊   | 6775/10000 [46:14<10:20,  5.20it/s]
 68%|██████▊   | 6776/10000 [46:14<10:11,  5.27it/s]
 68%|██████▊   | 6777/10000 [46:14<10:12,  5.26it/s]
 68%|██████▊   | 6778/10000 [46:14<10:08,  5.30it/s]
 68%|██████▊   | 6779/10000 [46:15<10:05,  5.32it/s]
 68%|██████▊   | 6780/10000 [46:15<10:02,  5.34it/s]
                                                    
{'loss': 0.0953, 'grad_norm': 0.6579843759536743, 'learning_rate': 2.5782017973986728e-05}

 68%|██████▊   | 6780/10000 [46:15<10:02,  5.34it/s]
 68%|██████▊   | 6781/10000 [46:15<10:01,  5.35it/s]
 68%|██████▊   | 6782/10000 [46:15<09:47,  5.47it/s]
 68%|██████▊   | 6783/10000 [46:15<09:53,  5.42it/s]
 68%|██████▊   | 6784/10000 [46:16<10:52,  4.93it/s]
 68%|██████▊   | 6785/10000 [46:16<10:30,  5.10it/s]
 68%|██████▊   | 6786/10000 [46:16<10:20,  5.18it/s]
 68%|██████▊   | 6787/10000 [46:16<10:10,  5.26it/s]
 68%|██████▊   | 6788/10000 [46:16<09:59,  5.36it/s]
 68%|██████▊   | 6789/10000 [46:16<10:01,  5.33it/s]
 68%|██████▊   | 6790/10000 [46:17<09:49,  5.44it/s]
                                                    
{'loss': 0.0821, 'grad_norm': 0.6659886240959167, 'learning_rate': 2.5637493713733374e-05}

 68%|██████▊   | 6790/10000 [46:17<09:49,  5.44it/s]
 68%|██████▊   | 6791/10000 [46:17<09:47,  5.46it/s]
 68%|██████▊   | 6792/10000 [46:17<09:44,  5.49it/s]
 68%|██████▊   | 6793/10000 [46:17<09:41,  5.52it/s]
 68%|██████▊   | 6794/10000 [46:17<09:35,  5.57it/s]Rank 0, Worker 2: Wait for shard 12 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 68%|██████▊   | 6795/10000 [46:18<09:50,  5.43it/s]
 68%|██████▊   | 6796/10000 [46:18<10:00,  5.33it/s]
 68%|██████▊   | 6797/10000 [46:18<10:04,  5.30it/s]
 68%|██████▊   | 6798/10000 [46:18<10:29,  5.08it/s]
 68%|██████▊   | 6799/10000 [46:18<10:29,  5.08it/s]
 68%|██████▊   | 6800/10000 [46:19<10:13,  5.22it/s]
                                                    
{'loss': 0.0898, 'grad_norm': 0.7826690077781677, 'learning_rate': 2.549323587794559e-05}

 68%|██████▊   | 6800/10000 [46:19<10:13,  5.22it/s]
 68%|██████▊   | 6801/10000 [46:19<10:10,  5.24it/s]
 68%|██████▊   | 6802/10000 [46:19<09:55,  5.37it/s]
 68%|██████▊   | 6803/10000 [46:19<09:52,  5.40it/s]
 68%|██████▊   | 6804/10000 [46:19<10:03,  5.30it/s]
 68%|██████▊   | 6805/10000 [46:19<10:08,  5.25it/s]
 68%|██████▊   | 6806/10000 [46:20<10:15,  5.19it/s]
 68%|██████▊   | 6807/10000 [46:20<10:18,  5.17it/s]
 68%|██████▊   | 6808/10000 [46:20<10:05,  5.27it/s]
 68%|██████▊   | 6809/10000 [46:20<09:53,  5.37it/s]
 68%|██████▊   | 6810/10000 [46:20<09:53,  5.38it/s]
                                                    
{'loss': 0.0879, 'grad_norm': 0.6394219398498535, 'learning_rate': 2.5349246044203895e-05}

 68%|██████▊   | 6810/10000 [46:20<09:53,  5.38it/s]
 68%|██████▊   | 6811/10000 [46:21<10:10,  5.23it/s]
 68%|██████▊   | 6812/10000 [46:21<10:05,  5.26it/s]
 68%|██████▊   | 6813/10000 [46:21<10:05,  5.26it/s]
 68%|██████▊   | 6814/10000 [46:21<10:11,  5.21it/s]
 68%|██████▊   | 6815/10000 [46:21<10:09,  5.22it/s]
 68%|██████▊   | 6816/10000 [46:22<10:15,  5.17it/s]
 68%|██████▊   | 6817/10000 [46:22<10:17,  5.15it/s]
 68%|██████▊   | 6818/10000 [46:22<10:21,  5.12it/s]
 68%|██████▊   | 6819/10000 [46:22<10:31,  5.04it/s]
 68%|██████▊   | 6820/10000 [46:22<10:31,  5.04it/s]
                                                    
{'loss': 0.0753, 'grad_norm': 0.7091806530952454, 'learning_rate': 2.520552578715808e-05}

 68%|██████▊   | 6820/10000 [46:22<10:31,  5.04it/s]Rank 0, Worker 4: Wait for shard 61 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 68%|██████▊   | 6821/10000 [46:23<10:41,  4.96it/s]
 68%|██████▊   | 6822/10000 [46:23<10:33,  5.01it/s]
 68%|██████▊   | 6823/10000 [46:23<10:35,  5.00it/s]
 68%|██████▊   | 6824/10000 [46:23<10:48,  4.90it/s]
 68%|██████▊   | 6825/10000 [46:23<10:37,  4.98it/s]
 68%|██████▊   | 6826/10000 [46:24<10:35,  4.99it/s]
 68%|██████▊   | 6827/10000 [46:24<10:32,  5.02it/s]
 68%|██████▊   | 6828/10000 [46:24<10:32,  5.01it/s]
 68%|██████▊   | 6829/10000 [46:24<10:28,  5.04it/s]
 68%|██████▊   | 6830/10000 [46:24<10:25,  5.07it/s]
                                                    
{'loss': 0.0748, 'grad_norm': 0.5920246839523315, 'learning_rate': 2.506207667850981e-05}

 68%|██████▊   | 6830/10000 [46:24<10:25,  5.07it/s]
 68%|██████▊   | 6831/10000 [46:25<10:34,  5.00it/s]
 68%|██████▊   | 6832/10000 [46:25<10:25,  5.06it/s]
 68%|██████▊   | 6833/10000 [46:25<10:28,  5.04it/s]
 68%|██████▊   | 6834/10000 [46:25<10:29,  5.03it/s]
 68%|██████▊   | 6835/10000 [46:25<10:28,  5.03it/s]
 68%|██████▊   | 6836/10000 [46:26<10:35,  4.98it/s]
 68%|██████▊   | 6837/10000 [46:26<10:32,  5.00it/s]
 68%|██████▊   | 6838/10000 [46:26<10:32,  5.00it/s]
 68%|██████▊   | 6839/10000 [46:26<10:25,  5.05it/s]
 68%|██████▊   | 6840/10000 [46:26<10:12,  5.16it/s]
                                                    
{'loss': 0.0834, 'grad_norm': 0.7445152997970581, 'learning_rate': 2.4918900286995555e-05}

 68%|██████▊   | 6840/10000 [46:26<10:12,  5.16it/s]
 68%|██████▊   | 6841/10000 [46:27<10:23,  5.07it/s]
 68%|██████▊   | 6842/10000 [46:27<10:15,  5.13it/s]
 68%|██████▊   | 6843/10000 [46:27<10:08,  5.19it/s]
 68%|██████▊   | 6844/10000 [46:27<10:18,  5.11it/s]
 68%|██████▊   | 6845/10000 [46:27<10:17,  5.11it/s]
 68%|██████▊   | 6846/10000 [46:28<10:14,  5.13it/s]
 68%|██████▊   | 6847/10000 [46:28<10:14,  5.13it/s]
 68%|██████▊   | 6848/10000 [46:28<10:16,  5.11it/s]
 68%|██████▊   | 6849/10000 [46:28<10:12,  5.14it/s]
 68%|██████▊   | 6850/10000 [46:28<10:13,  5.13it/s]
                                                    
{'loss': 0.073, 'grad_norm': 0.7014796137809753, 'learning_rate': 2.4775998178369458e-05}

 68%|██████▊   | 6850/10000 [46:28<10:13,  5.13it/s]
 69%|██████▊   | 6851/10000 [46:29<10:17,  5.10it/s]
 69%|██████▊   | 6852/10000 [46:29<10:01,  5.24it/s]
 69%|██████▊   | 6853/10000 [46:29<09:54,  5.29it/s]
 69%|██████▊   | 6854/10000 [46:29<09:52,  5.31it/s]
 69%|██████▊   | 6855/10000 [46:29<09:54,  5.29it/s]
 69%|██████▊   | 6856/10000 [46:29<10:03,  5.21it/s]
 69%|██████▊   | 6857/10000 [46:30<10:15,  5.11it/s]
 69%|██████▊   | 6858/10000 [46:30<10:27,  5.00it/s]
 69%|██████▊   | 6859/10000 [46:30<10:26,  5.01it/s]
 69%|██████▊   | 6860/10000 [46:30<10:28,  5.00it/s]
                                                    
{'loss': 0.0733, 'grad_norm': 0.6884494423866272, 'learning_rate': 2.4633371915386017e-05}

 69%|██████▊   | 6860/10000 [46:30<10:28,  5.00it/s]
 69%|██████▊   | 6861/10000 [46:30<10:34,  4.95it/s]
 69%|██████▊   | 6862/10000 [46:31<10:25,  5.02it/s]
 69%|██████▊   | 6863/10000 [46:31<10:21,  5.04it/s]
 69%|██████▊   | 6864/10000 [46:31<10:18,  5.07it/s]
 69%|██████▊   | 6865/10000 [46:31<10:17,  5.07it/s]
 69%|██████▊   | 6866/10000 [46:31<10:10,  5.14it/s]
 69%|██████▊   | 6867/10000 [46:32<10:09,  5.14it/s]
 69%|██████▊   | 6868/10000 [46:32<10:14,  5.10it/s]
 69%|██████▊   | 6869/10000 [46:32<10:15,  5.09it/s]
 69%|██████▊   | 6870/10000 [46:32<10:20,  5.04it/s]
                                                    
{'loss': 0.0829, 'grad_norm': 0.8311998248100281, 'learning_rate': 2.4491023057783235e-05}

 69%|██████▊   | 6870/10000 [46:32<10:20,  5.04it/s]
 69%|██████▊   | 6871/10000 [46:32<10:36,  4.91it/s]
 69%|██████▊   | 6872/10000 [46:33<10:15,  5.08it/s]
 69%|██████▊   | 6873/10000 [46:33<09:58,  5.23it/s]
 69%|██████▊   | 6874/10000 [46:33<09:47,  5.32it/s]
 69%|██████▉   | 6875/10000 [46:33<09:38,  5.41it/s]
 69%|██████▉   | 6876/10000 [46:33<09:38,  5.40it/s]
 69%|██████▉   | 6877/10000 [46:34<09:51,  5.28it/s]
 69%|██████▉   | 6878/10000 [46:34<10:00,  5.20it/s]
 69%|██████▉   | 6879/10000 [46:34<10:06,  5.15it/s]
 69%|██████▉   | 6880/10000 [46:34<10:06,  5.14it/s]
                                                    
{'loss': 0.0776, 'grad_norm': 0.8297330737113953, 'learning_rate': 2.4348953162265375e-05}

 69%|██████▉   | 6880/10000 [46:34<10:06,  5.14it/s]
 69%|██████▉   | 6881/10000 [46:34<10:06,  5.14it/s]
 69%|██████▉   | 6882/10000 [46:35<09:54,  5.24it/s]
 69%|██████▉   | 6883/10000 [46:35<09:50,  5.28it/s]
 69%|██████▉   | 6884/10000 [46:35<09:56,  5.22it/s]
 69%|██████▉   | 6885/10000 [46:35<10:14,  5.07it/s]
 69%|██████▉   | 6886/10000 [46:35<10:11,  5.09it/s]
 69%|██████▉   | 6887/10000 [46:36<10:03,  5.16it/s]
 69%|██████▉   | 6888/10000 [46:36<10:01,  5.17it/s]
 69%|██████▉   | 6889/10000 [46:36<09:43,  5.33it/s]
 69%|██████▉   | 6890/10000 [46:36<09:30,  5.45it/s]
                                                    
{'loss': 0.0704, 'grad_norm': 0.7187453508377075, 'learning_rate': 2.420716378248607e-05}

 69%|██████▉   | 6890/10000 [46:36<09:30,  5.45it/s]
 69%|██████▉   | 6891/10000 [46:36<09:34,  5.41it/s]
 69%|██████▉   | 6892/10000 [46:36<09:28,  5.46it/s]
 69%|██████▉   | 6893/10000 [46:37<09:24,  5.51it/s]
 69%|██████▉   | 6894/10000 [46:37<09:30,  5.44it/s]
 69%|██████▉   | 6895/10000 [46:37<09:34,  5.40it/s]
 69%|██████▉   | 6896/10000 [46:37<09:32,  5.42it/s]
 69%|██████▉   | 6897/10000 [46:37<09:35,  5.40it/s]
 69%|██████▉   | 6898/10000 [46:38<09:26,  5.48it/s]
 69%|██████▉   | 6899/10000 [46:38<09:16,  5.57it/s]
 69%|██████▉   | 6900/10000 [46:38<09:12,  5.61it/s]
                                                    
{'loss': 0.0749, 'grad_norm': 0.6742458343505859, 'learning_rate': 2.4065656469031266e-05}

 69%|██████▉   | 6900/10000 [46:38<09:12,  5.61it/s]
 69%|██████▉   | 6901/10000 [46:38<09:18,  5.55it/s]
 69%|██████▉   | 6902/10000 [46:38<09:20,  5.53it/s]
 69%|██████▉   | 6903/10000 [46:38<09:27,  5.46it/s]
 69%|██████▉   | 6904/10000 [46:39<09:32,  5.41it/s]
 69%|██████▉   | 6905/10000 [46:39<09:33,  5.40it/s]
 69%|██████▉   | 6906/10000 [46:39<09:32,  5.41it/s]
 69%|██████▉   | 6907/10000 [46:39<09:30,  5.42it/s]
 69%|██████▉   | 6908/10000 [46:39<09:26,  5.46it/s]
 69%|██████▉   | 6909/10000 [46:40<09:25,  5.47it/s]
 69%|██████▉   | 6910/10000 [46:40<10:00,  5.15it/s]
                                                    
{'loss': 0.062, 'grad_norm': 0.5884044170379639, 'learning_rate': 2.3924432769402268e-05}

 69%|██████▉   | 6910/10000 [46:40<10:00,  5.15it/s]
 69%|██████▉   | 6911/10000 [46:40<10:24,  4.95it/s]
 69%|██████▉   | 6912/10000 [46:40<10:20,  4.98it/s]
 69%|██████▉   | 6913/10000 [46:40<10:09,  5.06it/s]
 69%|██████▉   | 6914/10000 [46:41<09:53,  5.20it/s]
 69%|██████▉   | 6915/10000 [46:41<09:38,  5.33it/s]
 69%|██████▉   | 6916/10000 [46:41<09:29,  5.42it/s]
 69%|██████▉   | 6917/10000 [46:41<09:23,  5.47it/s]
 69%|██████▉   | 6918/10000 [46:41<09:28,  5.42it/s]
 69%|██████▉   | 6919/10000 [46:41<09:38,  5.32it/s]
 69%|██████▉   | 6920/10000 [46:42<09:40,  5.31it/s]
                                                    
{'loss': 0.0777, 'grad_norm': 0.5630632638931274, 'learning_rate': 2.3783494227998844e-05}

 69%|██████▉   | 6920/10000 [46:42<09:40,  5.31it/s]
 69%|██████▉   | 6921/10000 [46:42<09:48,  5.23it/s]
 69%|██████▉   | 6922/10000 [46:42<09:40,  5.30it/s]
 69%|██████▉   | 6923/10000 [46:42<09:28,  5.41it/s]Rank 0, Worker 5: Wait for shard 8 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 69%|██████▉   | 6924/10000 [46:42<09:48,  5.23it/s]
 69%|██████▉   | 6925/10000 [46:43<10:09,  5.05it/s]
 69%|██████▉   | 6926/10000 [46:43<11:03,  4.63it/s]
 69%|██████▉   | 6927/10000 [46:43<11:22,  4.50it/s]
 69%|██████▉   | 6928/10000 [46:43<11:30,  4.45it/s]
 69%|██████▉   | 6929/10000 [46:44<11:44,  4.36it/s]
 69%|██████▉   | 6930/10000 [46:44<11:41,  4.38it/s]
                                                    
{'loss': 0.0783, 'grad_norm': 0.7275298833847046, 'learning_rate': 2.3642842386102264e-05}

 69%|██████▉   | 6930/10000 [46:44<11:41,  4.38it/s]
 69%|██████▉   | 6931/10000 [46:44<11:18,  4.53it/s]
 69%|██████▉   | 6932/10000 [46:44<10:43,  4.76it/s]
 69%|██████▉   | 6933/10000 [46:44<10:34,  4.83it/s]Rank 0, Worker 3: Wait for shard 57 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 69%|██████▉   | 6934/10000 [46:45<10:30,  4.86it/s]
 69%|██████▉   | 6935/10000 [46:45<10:38,  4.80it/s]
 69%|██████▉   | 6936/10000 [46:45<10:44,  4.75it/s]
 69%|██████▉   | 6937/10000 [46:45<10:48,  4.72it/s]
 69%|██████▉   | 6938/10000 [46:45<10:55,  4.67it/s]
 69%|██████▉   | 6939/10000 [46:46<11:10,  4.57it/s]
 69%|██████▉   | 6940/10000 [46:46<10:52,  4.69it/s]
                                                    
{'loss': 0.0702, 'grad_norm': 0.7981938123703003, 'learning_rate': 2.3502478781858567e-05}

 69%|██████▉   | 6940/10000 [46:46<10:52,  4.69it/s]
 69%|██████▉   | 6941/10000 [46:46<11:06,  4.59it/s]
 69%|██████▉   | 6942/10000 [46:46<10:49,  4.71it/s]
 69%|██████▉   | 6943/10000 [46:47<10:39,  4.78it/s]
 69%|██████▉   | 6944/10000 [46:47<10:41,  4.76it/s]
 69%|██████▉   | 6945/10000 [46:47<10:47,  4.72it/s]
 69%|██████▉   | 6946/10000 [46:47<10:51,  4.69it/s]
 69%|██████▉   | 6947/10000 [46:47<11:09,  4.56it/s]
 69%|██████▉   | 6948/10000 [46:48<11:11,  4.55it/s]
 69%|██████▉   | 6949/10000 [46:48<11:28,  4.43it/s]
 70%|██████▉   | 6950/10000 [46:48<11:21,  4.48it/s]
                                                    
{'loss': 0.088, 'grad_norm': 0.7301228046417236, 'learning_rate': 2.3362404950261628e-05}

 70%|██████▉   | 6950/10000 [46:48<11:21,  4.48it/s]
 70%|██████▉   | 6951/10000 [46:48<11:07,  4.57it/s]
 70%|██████▉   | 6952/10000 [46:49<10:59,  4.62it/s]
 70%|██████▉   | 6953/10000 [46:49<10:56,  4.64it/s]
 70%|██████▉   | 6954/10000 [46:49<10:58,  4.63it/s]
 70%|██████▉   | 6955/10000 [46:49<10:53,  4.66it/s]
 70%|██████▉   | 6956/10000 [46:49<11:20,  4.47it/s]
 70%|██████▉   | 6957/10000 [46:50<11:38,  4.36it/s]
 70%|██████▉   | 6958/10000 [46:50<11:48,  4.29it/s]
 70%|██████▉   | 6959/10000 [46:50<11:37,  4.36it/s]
 70%|██████▉   | 6960/10000 [46:50<11:14,  4.51it/s]
                                                    
{'loss': 0.0724, 'grad_norm': 0.8947741985321045, 'learning_rate': 2.3222622423136458e-05}

 70%|██████▉   | 6960/10000 [46:50<11:14,  4.51it/s]
 70%|██████▉   | 6961/10000 [46:51<11:15,  4.50it/s]
 70%|██████▉   | 6962/10000 [46:51<11:18,  4.48it/s]
 70%|██████▉   | 6963/10000 [46:51<11:17,  4.48it/s]
 70%|██████▉   | 6964/10000 [46:51<11:36,  4.36it/s]
 70%|██████▉   | 6965/10000 [46:51<11:52,  4.26it/s]
 70%|██████▉   | 6966/10000 [46:52<11:50,  4.27it/s]
 70%|██████▉   | 6967/10000 [46:52<11:27,  4.41it/s]
 70%|██████▉   | 6968/10000 [46:52<11:06,  4.55it/s]
 70%|██████▉   | 6969/10000 [46:52<10:48,  4.67it/s]
 70%|██████▉   | 6970/10000 [46:53<10:38,  4.75it/s]
                                                    
{'loss': 0.0718, 'grad_norm': 0.5670645833015442, 'learning_rate': 2.3083132729122332e-05}

 70%|██████▉   | 6970/10000 [46:53<10:38,  4.75it/s]
 70%|██████▉   | 6971/10000 [46:53<10:44,  4.70it/s]
 70%|██████▉   | 6972/10000 [46:53<10:56,  4.61it/s]
 70%|██████▉   | 6973/10000 [46:53<11:22,  4.43it/s]
 70%|██████▉   | 6974/10000 [46:53<11:34,  4.36it/s]
 70%|██████▉   | 6975/10000 [46:54<11:16,  4.47it/s]
 70%|██████▉   | 6976/10000 [46:54<11:20,  4.45it/s]
 70%|██████▉   | 6977/10000 [46:54<11:21,  4.44it/s]
 70%|██████▉   | 6978/10000 [46:54<11:15,  4.48it/s]Rank 0, Worker 0: Wait for shard 56 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 70%|██████▉   | 6979/10000 [46:55<11:12,  4.49it/s]
 70%|██████▉   | 6980/10000 [46:55<11:25,  4.41it/s]
                                                    
{'loss': 0.0762, 'grad_norm': 0.6690959930419922, 'learning_rate': 2.294393739365621e-05}

 70%|██████▉   | 6980/10000 [46:55<11:25,  4.41it/s]
 70%|██████▉   | 6981/10000 [46:55<13:09,  3.82it/s]
 70%|██████▉   | 6982/10000 [46:55<14:07,  3.56it/s]
 70%|██████▉   | 6983/10000 [46:56<13:16,  3.79it/s]
 70%|██████▉   | 6984/10000 [46:56<12:31,  4.02it/s]
 70%|██████▉   | 6985/10000 [46:56<12:16,  4.09it/s]
 70%|██████▉   | 6986/10000 [46:56<11:48,  4.25it/s]
 70%|██████▉   | 6987/10000 [46:57<11:53,  4.22it/s]
 70%|██████▉   | 6988/10000 [46:57<12:23,  4.05it/s]
 70%|██████▉   | 6989/10000 [46:57<12:27,  4.03it/s]
 70%|██████▉   | 6990/10000 [46:57<12:15,  4.09it/s]
                                                    
{'loss': 0.0748, 'grad_norm': 0.649847686290741, 'learning_rate': 2.2805037938956e-05}

 70%|██████▉   | 6990/10000 [46:57<12:15,  4.09it/s]
 70%|██████▉   | 6991/10000 [46:58<11:53,  4.21it/s]
 70%|██████▉   | 6992/10000 [46:58<11:25,  4.39it/s]
 70%|██████▉   | 6993/10000 [46:58<11:03,  4.53it/s]
 70%|██████▉   | 6994/10000 [46:58<10:50,  4.62it/s]
 70%|██████▉   | 6995/10000 [46:58<11:02,  4.53it/s]
 70%|██████▉   | 6996/10000 [46:59<11:11,  4.47it/s]
 70%|██████▉   | 6997/10000 [46:59<11:04,  4.52it/s]
 70%|██████▉   | 6998/10000 [46:59<11:18,  4.43it/s]
 70%|██████▉   | 6999/10000 [46:59<11:02,  4.53it/s]
 70%|███████   | 7000/10000 [46:59<10:46,  4.64it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Caching shard...Rank 0, Worker 4: Caching shard...

Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 5: Caching shard...

Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.65 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.10 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.28 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.34 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.44 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.54 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 7000: eval/loss = 0.0879 (36s)

                                                    
{'loss': 0.0778, 'grad_norm': 0.6587846279144287, 'learning_rate': 2.266643588400386e-05}

 70%|███████   | 7000/10000 [47:36<10:46,  4.64it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-7000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-7000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-7000/wandb_config.json

 70%|███████   | 7001/10000 [48:04<16:20:19, 19.61s/it]
 70%|███████   | 7002/10000 [48:05<11:28:48, 13.79s/it]
 70%|███████   | 7003/10000 [48:05<8:04:43,  9.70s/it] 
 70%|███████   | 7004/10000 [48:05<5:42:17,  6.85s/it]
 70%|███████   | 7005/10000 [48:05<4:02:34,  4.86s/it]
 70%|███████   | 7006/10000 [48:05<2:52:53,  3.46s/it]
 70%|███████   | 7007/10000 [48:06<2:04:18,  2.49s/it]
 70%|███████   | 7008/10000 [48:06<1:29:55,  1.80s/it]
 70%|███████   | 7009/10000 [48:06<1:05:46,  1.32s/it]
 70%|███████   | 7010/10000 [48:06<48:40,  1.02it/s]  
                                                    
{'loss': 0.0747, 'grad_norm': 0.5851252675056458, 'learning_rate': 2.252813274452969e-05}

 70%|███████   | 7010/10000 [48:06<48:40,  1.02it/s]
 70%|███████   | 7011/10000 [48:06<37:03,  1.34it/s]
 70%|███████   | 7012/10000 [48:07<28:57,  1.72it/s]
 70%|███████   | 7013/10000 [48:07<23:24,  2.13it/s]
 70%|███████   | 7014/10000 [48:07<19:31,  2.55it/s]
 70%|███████   | 7015/10000 [48:07<16:47,  2.96it/s]
 70%|███████   | 7016/10000 [48:07<14:42,  3.38it/s]
 70%|███████   | 7017/10000 [48:08<13:17,  3.74it/s]
 70%|███████   | 7018/10000 [48:08<12:11,  4.07it/s]
 70%|███████   | 7019/10000 [48:08<11:24,  4.35it/s]
 70%|███████   | 7020/10000 [48:08<10:47,  4.60it/s]
                                                    
{'loss': 0.0725, 'grad_norm': 0.5657774806022644, 'learning_rate': 2.2390130032994427e-05}

 70%|███████   | 7020/10000 [48:08<10:47,  4.60it/s]
 70%|███████   | 7021/10000 [48:08<10:45,  4.61it/s]
 70%|███████   | 7022/10000 [48:09<10:34,  4.69it/s]
 70%|███████   | 7023/10000 [48:09<10:27,  4.74it/s]
 70%|███████   | 7024/10000 [48:09<10:25,  4.76it/s]
 70%|███████   | 7025/10000 [48:09<10:16,  4.83it/s]
 70%|███████   | 7026/10000 [48:09<10:07,  4.89it/s]
 70%|███████   | 7027/10000 [48:10<09:58,  4.97it/s]
 70%|███████   | 7028/10000 [48:10<10:23,  4.77it/s]
 70%|███████   | 7029/10000 [48:10<10:38,  4.65it/s]
 70%|███████   | 7030/10000 [48:10<10:46,  4.60it/s]
                                                    
{'loss': 0.0722, 'grad_norm': 0.7438356876373291, 'learning_rate': 2.2252429258573633e-05}

 70%|███████   | 7030/10000 [48:10<10:46,  4.60it/s]
 70%|███████   | 7031/10000 [48:11<11:30,  4.30it/s]
 70%|███████   | 7032/10000 [48:11<11:43,  4.22it/s]
 70%|███████   | 7033/10000 [48:11<11:25,  4.33it/s]
 70%|███████   | 7034/10000 [48:11<10:55,  4.52it/s]
 70%|███████   | 7035/10000 [48:11<10:34,  4.67it/s]
 70%|███████   | 7036/10000 [48:12<10:10,  4.85it/s]
 70%|███████   | 7037/10000 [48:12<09:55,  4.98it/s]
 70%|███████   | 7038/10000 [48:12<10:08,  4.87it/s]
 70%|███████   | 7039/10000 [48:12<10:16,  4.80it/s]
 70%|███████   | 7040/10000 [48:12<10:11,  4.84it/s]
                                                    
{'loss': 0.0754, 'grad_norm': 0.9332830905914307, 'learning_rate': 2.2115031927140904e-05}

 70%|███████   | 7040/10000 [48:12<10:11,  4.84it/s]
 70%|███████   | 7041/10000 [48:13<10:53,  4.53it/s]
 70%|███████   | 7042/10000 [48:13<10:41,  4.61it/s]
 70%|███████   | 7043/10000 [48:13<10:20,  4.77it/s]
 70%|███████   | 7044/10000 [48:13<09:59,  4.93it/s]
 70%|███████   | 7045/10000 [48:13<09:42,  5.07it/s]
 70%|███████   | 7046/10000 [48:14<09:27,  5.21it/s]
 70%|███████   | 7047/10000 [48:14<09:34,  5.14it/s]
 70%|███████   | 7048/10000 [48:14<09:56,  4.95it/s]
 70%|███████   | 7049/10000 [48:14<10:36,  4.64it/s]
 70%|███████   | 7050/10000 [48:14<10:29,  4.68it/s]
                                                    
{'loss': 0.0775, 'grad_norm': 0.8124223351478577, 'learning_rate': 2.1977939541251463e-05}

 70%|███████   | 7050/10000 [48:15<10:29,  4.68it/s]
 71%|███████   | 7051/10000 [48:15<10:37,  4.63it/s]
 71%|███████   | 7052/10000 [48:15<10:36,  4.63it/s]
 71%|███████   | 7053/10000 [48:15<10:19,  4.76it/s]
 71%|███████   | 7054/10000 [48:15<10:02,  4.89it/s]
 71%|███████   | 7055/10000 [48:15<09:44,  5.04it/s]
 71%|███████   | 7056/10000 [48:16<09:45,  5.03it/s]
 71%|███████   | 7057/10000 [48:16<10:00,  4.90it/s]
 71%|███████   | 7058/10000 [48:16<10:19,  4.75it/s]
 71%|███████   | 7059/10000 [48:16<10:48,  4.53it/s]
 71%|███████   | 7060/10000 [48:17<11:23,  4.30it/s]
                                                    
{'loss': 0.072, 'grad_norm': 0.7005982398986816, 'learning_rate': 2.1841153600125684e-05}

 71%|███████   | 7060/10000 [48:17<11:23,  4.30it/s]
 71%|███████   | 7061/10000 [48:17<11:36,  4.22it/s]
 71%|███████   | 7062/10000 [48:17<11:05,  4.41it/s]
 71%|███████   | 7063/10000 [48:17<10:43,  4.57it/s]
 71%|███████   | 7064/10000 [48:18<10:44,  4.56it/s]
 71%|███████   | 7065/10000 [48:18<10:37,  4.61it/s]
 71%|███████   | 7066/10000 [48:18<10:56,  4.47it/s]
 71%|███████   | 7067/10000 [48:18<11:50,  4.13it/s]
 71%|███████   | 7068/10000 [48:18<11:56,  4.09it/s]
 71%|███████   | 7069/10000 [48:19<11:58,  4.08it/s]
 71%|███████   | 7070/10000 [48:19<11:37,  4.20it/s]
                                                    
{'loss': 0.072, 'grad_norm': 0.7529388070106506, 'learning_rate': 2.170467559963267e-05}

 71%|███████   | 7070/10000 [48:19<11:37,  4.20it/s]
 71%|███████   | 7071/10000 [48:19<11:28,  4.25it/s]
 71%|███████   | 7072/10000 [48:19<11:09,  4.38it/s]
 71%|███████   | 7073/10000 [48:20<10:47,  4.52it/s]
 71%|███████   | 7074/10000 [48:20<10:38,  4.58it/s]
 71%|███████   | 7075/10000 [48:20<10:37,  4.59it/s]
 71%|███████   | 7076/10000 [48:20<10:39,  4.57it/s]
 71%|███████   | 7077/10000 [48:21<11:04,  4.40it/s]
 71%|███████   | 7078/10000 [48:21<11:05,  4.39it/s]
 71%|███████   | 7079/10000 [48:21<11:01,  4.42it/s]
 71%|███████   | 7080/10000 [48:21<10:59,  4.43it/s]
                                                    
{'loss': 0.0768, 'grad_norm': 0.7132534980773926, 'learning_rate': 2.1568507032273982e-05}

 71%|███████   | 7080/10000 [48:21<10:59,  4.43it/s]
 71%|███████   | 7081/10000 [48:21<10:55,  4.45it/s]
 71%|███████   | 7082/10000 [48:22<10:26,  4.66it/s]
 71%|███████   | 7083/10000 [48:22<10:16,  4.73it/s]
 71%|███████   | 7084/10000 [48:22<10:09,  4.78it/s]
 71%|███████   | 7085/10000 [48:22<10:18,  4.71it/s]
 71%|███████   | 7086/10000 [48:23<11:37,  4.18it/s]
 71%|███████   | 7087/10000 [48:23<11:40,  4.16it/s]
 71%|███████   | 7088/10000 [48:23<11:12,  4.33it/s]
 71%|███████   | 7089/10000 [48:23<11:07,  4.36it/s]
 71%|███████   | 7090/10000 [48:23<10:51,  4.47it/s]
                                                    
{'loss': 0.0845, 'grad_norm': 0.750144362449646, 'learning_rate': 2.1432649387167264e-05}

 71%|███████   | 7090/10000 [48:23<10:51,  4.47it/s]
 71%|███████   | 7091/10000 [48:24<11:00,  4.41it/s]
 71%|███████   | 7092/10000 [48:24<11:22,  4.26it/s]
 71%|███████   | 7093/10000 [48:24<11:55,  4.06it/s]
 71%|███████   | 7094/10000 [48:24<12:10,  3.98it/s]
 71%|███████   | 7095/10000 [48:25<11:46,  4.11it/s]
 71%|███████   | 7096/10000 [48:25<11:34,  4.18it/s]
 71%|███████   | 7097/10000 [48:25<11:17,  4.29it/s]
 71%|███████   | 7098/10000 [48:25<11:07,  4.35it/s]
 71%|███████   | 7099/10000 [48:26<10:55,  4.43it/s]
 71%|███████   | 7100/10000 [48:26<10:54,  4.43it/s]
                                                    
{'loss': 0.0932, 'grad_norm': 0.7026374340057373, 'learning_rate': 2.1297104150029973e-05}

 71%|███████   | 7100/10000 [48:26<10:54,  4.43it/s]
 71%|███████   | 7101/10000 [48:26<11:05,  4.36it/s]
 71%|███████   | 7102/10000 [48:26<10:47,  4.48it/s]
 71%|███████   | 7103/10000 [48:26<10:39,  4.53it/s]
 71%|███████   | 7104/10000 [48:27<10:30,  4.59it/s]
 71%|███████   | 7105/10000 [48:27<10:17,  4.69it/s]
 71%|███████   | 7106/10000 [48:27<09:56,  4.85it/s]
 71%|███████   | 7107/10000 [48:27<09:47,  4.92it/s]
 71%|███████   | 7108/10000 [48:27<09:50,  4.90it/s]
 71%|███████   | 7109/10000 [48:28<09:58,  4.83it/s]
 71%|███████   | 7110/10000 [48:28<11:44,  4.10it/s]
                                                    
{'loss': 0.0745, 'grad_norm': 0.8443405628204346, 'learning_rate': 2.116187280316307e-05}

 71%|███████   | 7110/10000 [48:28<11:44,  4.10it/s]
 71%|███████   | 7111/10000 [48:28<12:10,  3.95it/s]
 71%|███████   | 7112/10000 [48:28<12:01,  4.00it/s]
 71%|███████   | 7113/10000 [48:29<11:34,  4.16it/s]
 71%|███████   | 7114/10000 [48:29<10:59,  4.37it/s]
 71%|███████   | 7115/10000 [48:29<10:22,  4.64it/s]
 71%|███████   | 7116/10000 [48:29<10:03,  4.78it/s]
 71%|███████   | 7117/10000 [48:29<09:57,  4.83it/s]
 71%|███████   | 7118/10000 [48:30<09:56,  4.83it/s]
 71%|███████   | 7119/10000 [48:30<10:07,  4.74it/s]
 71%|███████   | 7120/10000 [48:30<10:14,  4.69it/s]
                                                    
{'loss': 0.0653, 'grad_norm': 0.6494272351264954, 'learning_rate': 2.1026956825434908e-05}

 71%|███████   | 7120/10000 [48:30<10:14,  4.69it/s]
 71%|███████   | 7121/10000 [48:30<10:47,  4.45it/s]
 71%|███████   | 7122/10000 [48:31<10:33,  4.55it/s]
 71%|███████   | 7123/10000 [48:31<10:17,  4.66it/s]
 71%|███████   | 7124/10000 [48:31<09:56,  4.82it/s]
 71%|███████▏  | 7125/10000 [48:31<09:47,  4.90it/s]
 71%|███████▏  | 7126/10000 [48:31<09:36,  4.98it/s]
 71%|███████▏  | 7127/10000 [48:32<09:42,  4.93it/s]
 71%|███████▏  | 7128/10000 [48:32<09:39,  4.95it/s]
 71%|███████▏  | 7129/10000 [48:32<09:47,  4.89it/s]
 71%|███████▏  | 7130/10000 [48:32<09:53,  4.84it/s]
                                                    
{'loss': 0.0817, 'grad_norm': 0.6059182286262512, 'learning_rate': 2.0892357692265017e-05}

 71%|███████▏  | 7130/10000 [48:32<09:53,  4.84it/s]
 71%|███████▏  | 7131/10000 [48:32<10:07,  4.72it/s]
 71%|███████▏  | 7132/10000 [48:33<10:00,  4.78it/s]
 71%|███████▏  | 7133/10000 [48:33<09:49,  4.86it/s]
 71%|███████▏  | 7134/10000 [48:33<09:24,  5.07it/s]
 71%|███████▏  | 7135/10000 [48:33<09:10,  5.21it/s]
 71%|███████▏  | 7136/10000 [48:33<09:26,  5.06it/s]
 71%|███████▏  | 7137/10000 [48:34<09:41,  4.92it/s]
 71%|███████▏  | 7138/10000 [48:34<09:59,  4.77it/s]
 71%|███████▏  | 7139/10000 [48:34<10:32,  4.53it/s]
 71%|███████▏  | 7140/10000 [48:34<10:26,  4.57it/s]
                                                    
{'loss': 0.0822, 'grad_norm': 0.7888148427009583, 'learning_rate': 2.0758076875607947e-05}

 71%|███████▏  | 7140/10000 [48:34<10:26,  4.57it/s]
 71%|███████▏  | 7141/10000 [48:35<10:32,  4.52it/s]
 71%|███████▏  | 7142/10000 [48:35<10:04,  4.73it/s]
 71%|███████▏  | 7143/10000 [48:35<09:38,  4.94it/s]
 71%|███████▏  | 7144/10000 [48:35<09:48,  4.85it/s]
 71%|███████▏  | 7145/10000 [48:35<09:42,  4.91it/s]
 71%|███████▏  | 7146/10000 [48:36<09:44,  4.89it/s]
 71%|███████▏  | 7147/10000 [48:36<10:10,  4.67it/s]
 71%|███████▏  | 7148/10000 [48:36<10:10,  4.67it/s]
 71%|███████▏  | 7149/10000 [48:36<09:57,  4.77it/s]
 72%|███████▏  | 7150/10000 [48:36<09:40,  4.91it/s]
                                                    
{'loss': 0.0748, 'grad_norm': 0.727286696434021, 'learning_rate': 2.0624115843937207e-05}

 72%|███████▏  | 7150/10000 [48:36<09:40,  4.91it/s]
 72%|███████▏  | 7151/10000 [48:37<09:32,  4.98it/s]
 72%|███████▏  | 7152/10000 [48:37<09:35,  4.95it/s]
 72%|███████▏  | 7153/10000 [48:37<09:56,  4.77it/s]
 72%|███████▏  | 7154/10000 [48:37<09:56,  4.77it/s]
 72%|███████▏  | 7155/10000 [48:37<10:01,  4.73it/s]
 72%|███████▏  | 7156/10000 [48:38<10:01,  4.73it/s]
 72%|███████▏  | 7157/10000 [48:38<09:55,  4.77it/s]
 72%|███████▏  | 7158/10000 [48:38<09:31,  4.97it/s]
 72%|███████▏  | 7159/10000 [48:38<09:10,  5.16it/s]
 72%|███████▏  | 7160/10000 [48:38<09:02,  5.23it/s]
                                                    
{'loss': 0.092, 'grad_norm': 0.7251013517379761, 'learning_rate': 2.0490476062229157e-05}

 72%|███████▏  | 7160/10000 [48:38<09:02,  5.23it/s]
 72%|███████▏  | 7161/10000 [48:39<09:16,  5.11it/s]
 72%|███████▏  | 7162/10000 [48:39<09:18,  5.08it/s]
 72%|███████▏  | 7163/10000 [48:39<09:22,  5.05it/s]
 72%|███████▏  | 7164/10000 [48:39<09:26,  5.01it/s]
 72%|███████▏  | 7165/10000 [48:39<09:35,  4.92it/s]
 72%|███████▏  | 7166/10000 [48:40<09:34,  4.93it/s]
 72%|███████▏  | 7167/10000 [48:40<09:28,  4.98it/s]
 72%|███████▏  | 7168/10000 [48:40<09:14,  5.11it/s]
 72%|███████▏  | 7169/10000 [48:40<08:59,  5.24it/s]
 72%|███████▏  | 7170/10000 [48:40<09:00,  5.24it/s]
                                                    
{'loss': 0.1079, 'grad_norm': 0.9286296963691711, 'learning_rate': 2.035715899194704e-05}

 72%|███████▏  | 7170/10000 [48:40<09:00,  5.24it/s]
 72%|███████▏  | 7171/10000 [48:41<09:18,  5.06it/s]
 72%|███████▏  | 7172/10000 [48:41<09:28,  4.97it/s]
 72%|███████▏  | 7173/10000 [48:41<09:37,  4.90it/s]
 72%|███████▏  | 7174/10000 [48:41<09:44,  4.84it/s]
 72%|███████▏  | 7175/10000 [48:41<09:36,  4.90it/s]
 72%|███████▏  | 7176/10000 [48:42<09:29,  4.96it/s]
 72%|███████▏  | 7177/10000 [48:42<09:16,  5.08it/s]
 72%|███████▏  | 7178/10000 [48:42<09:19,  5.05it/s]
 72%|███████▏  | 7179/10000 [48:42<09:20,  5.03it/s]
 72%|███████▏  | 7180/10000 [48:42<09:20,  5.03it/s]
                                                    
{'loss': 0.0808, 'grad_norm': 0.74485182762146, 'learning_rate': 2.022416609102499e-05}

 72%|███████▏  | 7180/10000 [48:42<09:20,  5.03it/s]
 72%|███████▏  | 7181/10000 [48:43<09:28,  4.95it/s]
 72%|███████▏  | 7182/10000 [48:43<09:22,  5.01it/s]
 72%|███████▏  | 7183/10000 [48:43<09:12,  5.10it/s]
 72%|███████▏  | 7184/10000 [48:43<09:11,  5.11it/s]
 72%|███████▏  | 7185/10000 [48:43<09:13,  5.09it/s]
 72%|███████▏  | 7186/10000 [48:44<09:12,  5.09it/s]
 72%|███████▏  | 7187/10000 [48:44<09:14,  5.08it/s]
 72%|███████▏  | 7188/10000 [48:44<09:07,  5.14it/s]
 72%|███████▏  | 7189/10000 [48:44<08:55,  5.25it/s]
 72%|███████▏  | 7190/10000 [48:44<08:43,  5.36it/s]
                                                    
{'loss': 0.0882, 'grad_norm': 0.6243143081665039, 'learning_rate': 2.009149881385205e-05}

 72%|███████▏  | 7190/10000 [48:44<08:43,  5.36it/s]
 72%|███████▏  | 7191/10000 [48:44<08:44,  5.36it/s]
 72%|███████▏  | 7192/10000 [48:45<08:38,  5.41it/s]
 72%|███████▏  | 7193/10000 [48:45<08:53,  5.27it/s]
 72%|███████▏  | 7194/10000 [48:45<09:06,  5.14it/s]
 72%|███████▏  | 7195/10000 [48:45<09:23,  4.98it/s]
 72%|███████▏  | 7196/10000 [48:45<09:15,  5.04it/s]
 72%|███████▏  | 7197/10000 [48:46<08:56,  5.22it/s]
 72%|███████▏  | 7198/10000 [48:46<08:40,  5.38it/s]
 72%|███████▏  | 7199/10000 [48:46<08:33,  5.46it/s]
 72%|███████▏  | 7200/10000 [48:46<08:36,  5.42it/s]
                                                    
{'loss': 0.0669, 'grad_norm': 0.5934269428253174, 'learning_rate': 1.995915861125634e-05}

 72%|███████▏  | 7200/10000 [48:46<08:36,  5.42it/s]
 72%|███████▏  | 7201/10000 [48:46<08:53,  5.25it/s]
 72%|███████▏  | 7202/10000 [48:47<08:59,  5.19it/s]
 72%|███████▏  | 7203/10000 [48:47<09:08,  5.10it/s]
 72%|███████▏  | 7204/10000 [48:47<09:15,  5.04it/s]
 72%|███████▏  | 7205/10000 [48:47<09:15,  5.03it/s]
 72%|███████▏  | 7206/10000 [48:47<08:54,  5.22it/s]
 72%|███████▏  | 7207/10000 [48:48<08:39,  5.38it/s]
 72%|███████▏  | 7208/10000 [48:48<08:33,  5.44it/s]
 72%|███████▏  | 7209/10000 [48:48<08:28,  5.49it/s]
 72%|███████▏  | 7210/10000 [48:48<08:38,  5.38it/s]
                                                    
{'loss': 0.076, 'grad_norm': 0.673790693283081, 'learning_rate': 1.9827146930489065e-05}

 72%|███████▏  | 7210/10000 [48:48<08:38,  5.38it/s]
 72%|███████▏  | 7211/10000 [48:48<08:54,  5.22it/s]
 72%|███████▏  | 7212/10000 [48:49<08:58,  5.18it/s]
 72%|███████▏  | 7213/10000 [48:49<08:52,  5.24it/s]
 72%|███████▏  | 7214/10000 [48:49<08:47,  5.29it/s]
 72%|███████▏  | 7215/10000 [48:49<08:36,  5.39it/s]
 72%|███████▏  | 7216/10000 [48:49<08:29,  5.47it/s]
 72%|███████▏  | 7217/10000 [48:49<08:24,  5.52it/s]
 72%|███████▏  | 7218/10000 [48:50<08:24,  5.52it/s]
 72%|███████▏  | 7219/10000 [48:50<08:32,  5.42it/s]
 72%|███████▏  | 7220/10000 [48:50<08:39,  5.35it/s]
                                                    
{'loss': 0.0916, 'grad_norm': 0.9269189834594727, 'learning_rate': 1.9695465215208848e-05}

 72%|███████▏  | 7220/10000 [48:50<08:39,  5.35it/s]
 72%|███████▏  | 7221/10000 [48:50<09:03,  5.11it/s]
 72%|███████▏  | 7222/10000 [48:50<09:02,  5.12it/s]
 72%|███████▏  | 7223/10000 [48:51<09:00,  5.14it/s]
 72%|███████▏  | 7224/10000 [48:51<08:44,  5.29it/s]
 72%|███████▏  | 7225/10000 [48:51<08:31,  5.43it/s]
 72%|███████▏  | 7226/10000 [48:51<08:27,  5.47it/s]
 72%|███████▏  | 7227/10000 [48:51<08:22,  5.52it/s]
 72%|███████▏  | 7228/10000 [48:51<08:17,  5.57it/s]
 72%|███████▏  | 7229/10000 [48:52<08:24,  5.49it/s]
 72%|███████▏  | 7230/10000 [48:52<08:28,  5.45it/s]
                                                    
{'loss': 0.0751, 'grad_norm': 0.6495062708854675, 'learning_rate': 1.9564114905465813e-05}

 72%|███████▏  | 7230/10000 [48:52<08:28,  5.45it/s]
 72%|███████▏  | 7231/10000 [48:52<08:39,  5.33it/s]
 72%|███████▏  | 7232/10000 [48:52<08:47,  5.25it/s]
 72%|███████▏  | 7233/10000 [48:52<08:44,  5.27it/s]
 72%|███████▏  | 7234/10000 [48:53<08:40,  5.31it/s]
 72%|███████▏  | 7235/10000 [48:53<08:36,  5.35it/s]
 72%|███████▏  | 7236/10000 [48:53<08:32,  5.39it/s]
 72%|███████▏  | 7237/10000 [48:53<08:29,  5.42it/s]
 72%|███████▏  | 7238/10000 [48:53<08:25,  5.46it/s]
 72%|███████▏  | 7239/10000 [48:54<08:27,  5.44it/s]
 72%|███████▏  | 7240/10000 [48:54<08:28,  5.43it/s]
                                                    
{'loss': 0.0847, 'grad_norm': 0.7508002519607544, 'learning_rate': 1.9433097437685936e-05}

 72%|███████▏  | 7240/10000 [48:54<08:28,  5.43it/s]
 72%|███████▏  | 7241/10000 [48:54<08:42,  5.28it/s]
 72%|███████▏  | 7242/10000 [48:54<08:48,  5.22it/s]
 72%|███████▏  | 7243/10000 [48:54<08:45,  5.25it/s]
 72%|███████▏  | 7244/10000 [48:54<08:36,  5.34it/s]
 72%|███████▏  | 7245/10000 [48:55<08:28,  5.42it/s]
 72%|███████▏  | 7246/10000 [48:55<08:22,  5.49it/s]
 72%|███████▏  | 7247/10000 [48:55<08:22,  5.48it/s]
 72%|███████▏  | 7248/10000 [48:55<08:29,  5.41it/s]
 72%|███████▏  | 7249/10000 [48:55<08:31,  5.38it/s]
 72%|███████▎  | 7250/10000 [48:56<08:31,  5.38it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 3: Caching shard...

Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.44 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 16.77 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.95 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.16 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.20 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.29 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 7250: eval/loss = 0.0981 (35s)

                                                    
{'loss': 0.0694, 'grad_norm': 0.5888226628303528, 'learning_rate': 1.930241424465521e-05}

 72%|███████▎  | 7250/10000 [49:31<08:31,  5.38it/s]
 73%|███████▎  | 7251/10000 [49:31<8:13:25, 10.77s/it]
 73%|███████▎  | 7252/10000 [49:31<5:47:47,  7.59s/it]
 73%|███████▎  | 7253/10000 [49:31<4:05:55,  5.37s/it]
 73%|███████▎  | 7254/10000 [49:32<2:54:32,  3.81s/it]
 73%|███████▎  | 7255/10000 [49:32<2:04:33,  2.72s/it]
 73%|███████▎  | 7256/10000 [49:32<1:29:42,  1.96s/it]
 73%|███████▎  | 7257/10000 [49:32<1:05:25,  1.43s/it]
 73%|███████▎  | 7258/10000 [49:32<48:43,  1.07s/it]  
 73%|███████▎  | 7259/10000 [49:33<37:01,  1.23it/s]
 73%|███████▎  | 7260/10000 [49:33<28:57,  1.58it/s]
                                                    
{'loss': 0.0813, 'grad_norm': 0.7730762362480164, 'learning_rate': 1.9172066755504115e-05}

 73%|███████▎  | 7260/10000 [49:33<28:57,  1.58it/s]
 73%|███████▎  | 7261/10000 [49:33<23:19,  1.96it/s]
 73%|███████▎  | 7262/10000 [49:33<19:13,  2.37it/s]
 73%|███████▎  | 7263/10000 [49:33<16:15,  2.81it/s]
 73%|███████▎  | 7264/10000 [49:34<14:13,  3.21it/s]
 73%|███████▎  | 7265/10000 [49:34<12:47,  3.57it/s]
 73%|███████▎  | 7266/10000 [49:34<11:45,  3.88it/s]
 73%|███████▎  | 7267/10000 [49:34<11:09,  4.08it/s]
 73%|███████▎  | 7268/10000 [49:34<10:42,  4.25it/s]
 73%|███████▎  | 7269/10000 [49:35<10:22,  4.39it/s]
 73%|███████▎  | 7270/10000 [49:35<09:46,  4.65it/s]
                                                    
{'loss': 0.0625, 'grad_norm': 0.6258568167686462, 'learning_rate': 1.9042056395691914e-05}

 73%|███████▎  | 7270/10000 [49:35<09:46,  4.65it/s]
 73%|███████▎  | 7271/10000 [49:35<09:33,  4.76it/s]
 73%|███████▎  | 7272/10000 [49:35<09:13,  4.93it/s]
 73%|███████▎  | 7273/10000 [49:35<09:03,  5.01it/s]
 73%|███████▎  | 7274/10000 [49:36<08:52,  5.12it/s]
 73%|███████▎  | 7275/10000 [49:36<08:35,  5.28it/s]
 73%|███████▎  | 7276/10000 [49:36<08:26,  5.37it/s]
 73%|███████▎  | 7277/10000 [49:36<08:17,  5.48it/s]
 73%|███████▎  | 7278/10000 [49:36<08:21,  5.43it/s]
 73%|███████▎  | 7279/10000 [49:37<08:29,  5.34it/s]
 73%|███████▎  | 7280/10000 [49:37<09:06,  4.97it/s]
                                                    
{'loss': 0.0768, 'grad_norm': 0.6968636512756348, 'learning_rate': 1.8912384586991066e-05}

 73%|███████▎  | 7280/10000 [49:37<09:06,  4.97it/s]
 73%|███████▎  | 7281/10000 [49:37<09:19,  4.86it/s]
 73%|███████▎  | 7282/10000 [49:37<09:13,  4.91it/s]
 73%|███████▎  | 7283/10000 [49:37<09:12,  4.92it/s]
 73%|███████▎  | 7284/10000 [49:38<08:58,  5.04it/s]
 73%|███████▎  | 7285/10000 [49:38<08:46,  5.15it/s]
 73%|███████▎  | 7286/10000 [49:38<08:31,  5.30it/s]
 73%|███████▎  | 7287/10000 [49:38<08:46,  5.15it/s]
 73%|███████▎  | 7288/10000 [49:38<08:56,  5.06it/s]
 73%|███████▎  | 7289/10000 [49:39<09:17,  4.86it/s]
 73%|███████▎  | 7290/10000 [49:39<09:41,  4.66it/s]
                                                    
{'loss': 0.0702, 'grad_norm': 0.707789957523346, 'learning_rate': 1.8783052747471717e-05}

 73%|███████▎  | 7290/10000 [49:39<09:41,  4.66it/s]
 73%|███████▎  | 7291/10000 [49:39<09:41,  4.66it/s]
 73%|███████▎  | 7292/10000 [49:39<09:31,  4.74it/s]
 73%|███████▎  | 7293/10000 [49:39<09:19,  4.84it/s]
 73%|███████▎  | 7294/10000 [49:40<09:15,  4.87it/s]
 73%|███████▎  | 7295/10000 [49:40<09:03,  4.98it/s]
 73%|███████▎  | 7296/10000 [49:40<09:02,  4.99it/s]
 73%|███████▎  | 7297/10000 [49:40<09:17,  4.85it/s]
 73%|███████▎  | 7298/10000 [49:40<09:14,  4.87it/s]
 73%|███████▎  | 7299/10000 [49:41<09:28,  4.75it/s]
 73%|███████▎  | 7300/10000 [49:41<09:29,  4.74it/s]
                                                    
{'loss': 0.0738, 'grad_norm': 0.9817585349082947, 'learning_rate': 1.865406229148611e-05}

 73%|███████▎  | 7300/10000 [49:41<09:29,  4.74it/s]
 73%|███████▎  | 7301/10000 [49:41<09:08,  4.92it/s]
 73%|███████▎  | 7302/10000 [49:41<08:52,  5.07it/s]
 73%|███████▎  | 7303/10000 [49:41<08:39,  5.19it/s]
 73%|███████▎  | 7304/10000 [49:42<08:24,  5.34it/s]
 73%|███████▎  | 7305/10000 [49:42<08:23,  5.35it/s]
 73%|███████▎  | 7306/10000 [49:42<08:23,  5.35it/s]
 73%|███████▎  | 7307/10000 [49:42<08:26,  5.32it/s]
 73%|███████▎  | 7308/10000 [49:42<08:26,  5.32it/s]
 73%|███████▎  | 7309/10000 [49:43<08:32,  5.25it/s]
 73%|███████▎  | 7310/10000 [49:43<08:23,  5.34it/s]
                                                    
{'loss': 0.0816, 'grad_norm': 0.7625161409378052, 'learning_rate': 1.8525414629653233e-05}

 73%|███████▎  | 7310/10000 [49:43<08:23,  5.34it/s]
 73%|███████▎  | 7311/10000 [49:43<08:28,  5.29it/s]
 73%|███████▎  | 7312/10000 [49:43<08:21,  5.36it/s]
 73%|███████▎  | 7313/10000 [49:43<08:14,  5.44it/s]
 73%|███████▎  | 7314/10000 [49:43<08:15,  5.42it/s]
 73%|███████▎  | 7315/10000 [49:44<08:14,  5.43it/s]
 73%|███████▎  | 7316/10000 [49:44<08:14,  5.43it/s]
 73%|███████▎  | 7317/10000 [49:44<08:22,  5.33it/s]
 73%|███████▎  | 7318/10000 [49:44<08:13,  5.43it/s]
 73%|███████▎  | 7319/10000 [49:44<08:06,  5.51it/s]
 73%|███████▎  | 7320/10000 [49:45<08:02,  5.55it/s]
                                                    
{'loss': 0.0829, 'grad_norm': 0.5806397199630737, 'learning_rate': 1.8397111168843255e-05}

 73%|███████▎  | 7320/10000 [49:45<08:02,  5.55it/s]
 73%|███████▎  | 7321/10000 [49:45<08:07,  5.49it/s]
 73%|███████▎  | 7322/10000 [49:45<08:37,  5.17it/s]
 73%|███████▎  | 7323/10000 [49:45<08:31,  5.23it/s]
 73%|███████▎  | 7324/10000 [49:45<08:27,  5.27it/s]
 73%|███████▎  | 7325/10000 [49:46<08:17,  5.37it/s]
 73%|███████▎  | 7326/10000 [49:46<08:14,  5.41it/s]
 73%|███████▎  | 7327/10000 [49:46<08:06,  5.50it/s]
 73%|███████▎  | 7328/10000 [49:46<08:10,  5.45it/s]
 73%|███████▎  | 7329/10000 [49:46<08:15,  5.39it/s]
 73%|███████▎  | 7330/10000 [49:46<08:16,  5.38it/s]
                                                    
{'loss': 0.0836, 'grad_norm': 0.7508918642997742, 'learning_rate': 1.8269153312162323e-05}

 73%|███████▎  | 7330/10000 [49:47<08:16,  5.38it/s]
 73%|███████▎  | 7331/10000 [49:47<10:55,  4.07it/s]
 73%|███████▎  | 7332/10000 [49:47<10:01,  4.44it/s]
 73%|███████▎  | 7333/10000 [49:47<09:23,  4.74it/s]
 73%|███████▎  | 7334/10000 [49:47<08:57,  4.96it/s]
 73%|███████▎  | 7335/10000 [49:48<08:35,  5.17it/s]
 73%|███████▎  | 7336/10000 [49:48<08:23,  5.30it/s]
 73%|███████▎  | 7337/10000 [49:48<08:24,  5.28it/s]
 73%|███████▎  | 7338/10000 [49:48<08:34,  5.17it/s]
 73%|███████▎  | 7339/10000 [49:48<08:38,  5.13it/s]
 73%|███████▎  | 7340/10000 [49:49<08:37,  5.14it/s]
                                                    
{'loss': 0.0714, 'grad_norm': 0.6141926050186157, 'learning_rate': 1.8141542458937054e-05}

 73%|███████▎  | 7340/10000 [49:49<08:37,  5.14it/s]
 73%|███████▎  | 7341/10000 [49:49<08:33,  5.18it/s]
 73%|███████▎  | 7342/10000 [49:49<08:19,  5.32it/s]
 73%|███████▎  | 7343/10000 [49:49<08:12,  5.39it/s]
 73%|███████▎  | 7344/10000 [49:49<08:05,  5.47it/s]
 73%|███████▎  | 7345/10000 [49:49<08:04,  5.48it/s]
 73%|███████▎  | 7346/10000 [49:50<08:09,  5.42it/s]
 73%|███████▎  | 7347/10000 [49:50<08:07,  5.44it/s]
 73%|███████▎  | 7348/10000 [49:50<08:01,  5.51it/s]
 73%|███████▎  | 7349/10000 [49:50<07:56,  5.56it/s]
 74%|███████▎  | 7350/10000 [49:50<07:53,  5.60it/s]
                                                    
{'loss': 0.0627, 'grad_norm': 0.6709929704666138, 'learning_rate': 1.8014280004699268e-05}

 74%|███████▎  | 7350/10000 [49:50<07:53,  5.60it/s]
 74%|███████▎  | 7351/10000 [49:50<07:56,  5.56it/s]
 74%|███████▎  | 7352/10000 [49:51<07:54,  5.58it/s]
 74%|███████▎  | 7353/10000 [49:51<07:51,  5.61it/s]
 74%|███████▎  | 7354/10000 [49:51<07:45,  5.68it/s]
 74%|███████▎  | 7355/10000 [49:51<07:49,  5.63it/s]
 74%|███████▎  | 7356/10000 [49:51<07:55,  5.56it/s]
 74%|███████▎  | 7357/10000 [49:52<07:58,  5.52it/s]
 74%|███████▎  | 7358/10000 [49:52<08:07,  5.42it/s]
 74%|███████▎  | 7359/10000 [49:52<08:11,  5.37it/s]
 74%|███████▎  | 7360/10000 [49:52<08:08,  5.40it/s]
                                                    
{'loss': 0.0653, 'grad_norm': 0.6348344087600708, 'learning_rate': 1.788736734117078e-05}

 74%|███████▎  | 7360/10000 [49:52<08:08,  5.40it/s]
 74%|███████▎  | 7361/10000 [49:52<08:10,  5.39it/s]
 74%|███████▎  | 7362/10000 [49:52<08:01,  5.48it/s]
 74%|███████▎  | 7363/10000 [49:53<07:54,  5.56it/s]
 74%|███████▎  | 7364/10000 [49:53<07:51,  5.60it/s]
 74%|███████▎  | 7365/10000 [49:53<07:55,  5.55it/s]
 74%|███████▎  | 7366/10000 [49:53<08:10,  5.37it/s]
 74%|███████▎  | 7367/10000 [49:53<08:36,  5.09it/s]
 74%|███████▎  | 7368/10000 [49:54<08:34,  5.12it/s]
 74%|███████▎  | 7369/10000 [49:54<08:19,  5.27it/s]
 74%|███████▎  | 7370/10000 [49:54<08:11,  5.35it/s]
                                                    
{'loss': 0.0814, 'grad_norm': 0.7334785461425781, 'learning_rate': 1.7760805856248152e-05}

 74%|███████▎  | 7370/10000 [49:54<08:11,  5.35it/s]
 74%|███████▎  | 7371/10000 [49:54<08:15,  5.31it/s]
 74%|███████▎  | 7372/10000 [49:54<08:16,  5.30it/s]
 74%|███████▎  | 7373/10000 [49:55<08:16,  5.29it/s]
 74%|███████▎  | 7374/10000 [49:55<08:21,  5.23it/s]
 74%|███████▍  | 7375/10000 [49:55<08:17,  5.28it/s]
 74%|███████▍  | 7376/10000 [49:55<08:15,  5.29it/s]
 74%|███████▍  | 7377/10000 [49:55<08:05,  5.40it/s]
 74%|███████▍  | 7378/10000 [49:55<08:02,  5.44it/s]
 74%|███████▍  | 7379/10000 [49:56<08:04,  5.41it/s]
 74%|███████▍  | 7380/10000 [49:56<08:08,  5.37it/s]
                                                    
{'loss': 0.0705, 'grad_norm': 0.7779228091239929, 'learning_rate': 1.7634596933987518e-05}

 74%|███████▍  | 7380/10000 [49:56<08:08,  5.37it/s]
 74%|███████▍  | 7381/10000 [49:56<08:26,  5.17it/s]
 74%|███████▍  | 7382/10000 [49:56<08:22,  5.21it/s]
 74%|███████▍  | 7383/10000 [49:56<08:11,  5.33it/s]
 74%|███████▍  | 7384/10000 [49:57<08:11,  5.32it/s]
 74%|███████▍  | 7385/10000 [49:57<08:02,  5.41it/s]
 74%|███████▍  | 7386/10000 [49:57<07:57,  5.48it/s]
 74%|███████▍  | 7387/10000 [49:57<07:53,  5.52it/s]
 74%|███████▍  | 7388/10000 [49:57<07:59,  5.44it/s]
 74%|███████▍  | 7389/10000 [49:58<08:06,  5.37it/s]
 74%|███████▍  | 7390/10000 [49:58<08:18,  5.24it/s]
                                                    
{'loss': 0.0678, 'grad_norm': 0.7552506327629089, 'learning_rate': 1.7508741954589404e-05}

 74%|███████▍  | 7390/10000 [49:58<08:18,  5.24it/s]
 74%|███████▍  | 7391/10000 [49:58<08:24,  5.17it/s]
 74%|███████▍  | 7392/10000 [49:58<08:20,  5.21it/s]
 74%|███████▍  | 7393/10000 [49:59<11:52,  3.66it/s]
 74%|███████▍  | 7394/10000 [49:59<10:56,  3.97it/s]
 74%|███████▍  | 7395/10000 [49:59<09:59,  4.34it/s]
 74%|███████▍  | 7396/10000 [49:59<09:20,  4.65it/s]
 74%|███████▍  | 7397/10000 [49:59<09:00,  4.82it/s]
 74%|███████▍  | 7398/10000 [50:00<08:42,  4.98it/s]
 74%|███████▍  | 7399/10000 [50:00<08:49,  4.91it/s]
 74%|███████▍  | 7400/10000 [50:00<08:52,  4.88it/s]
                                                    
{'loss': 0.0763, 'grad_norm': 0.6790664792060852, 'learning_rate': 1.7383242294383717e-05}

 74%|███████▍  | 7400/10000 [50:00<08:52,  4.88it/s]
 74%|███████▍  | 7401/10000 [50:00<08:50,  4.90it/s]
 74%|███████▍  | 7402/10000 [50:00<08:55,  4.85it/s]
 74%|███████▍  | 7403/10000 [50:01<08:57,  4.83it/s]
 74%|███████▍  | 7404/10000 [50:01<09:07,  4.74it/s]
 74%|███████▍  | 7405/10000 [50:01<09:11,  4.70it/s]
 74%|███████▍  | 7406/10000 [50:01<09:19,  4.63it/s]
 74%|███████▍  | 7407/10000 [50:01<09:21,  4.61it/s]
 74%|███████▍  | 7408/10000 [50:02<09:20,  4.62it/s]
 74%|███████▍  | 7409/10000 [50:02<09:19,  4.63it/s]
 74%|███████▍  | 7410/10000 [50:02<09:04,  4.76it/s]
                                                    
{'loss': 0.0607, 'grad_norm': 0.6178010702133179, 'learning_rate': 1.7258099325814632e-05}

 74%|███████▍  | 7410/10000 [50:02<09:04,  4.76it/s]
 74%|███████▍  | 7411/10000 [50:02<09:03,  4.77it/s]
 74%|███████▍  | 7412/10000 [50:02<08:56,  4.83it/s]
 74%|███████▍  | 7413/10000 [50:03<08:41,  4.96it/s]
 74%|███████▍  | 7414/10000 [50:03<08:33,  5.04it/s]
 74%|███████▍  | 7415/10000 [50:03<08:28,  5.08it/s]
 74%|███████▍  | 7416/10000 [50:03<08:39,  4.97it/s]
 74%|███████▍  | 7417/10000 [50:04<09:13,  4.66it/s]
 74%|███████▍  | 7418/10000 [50:04<09:27,  4.55it/s]
 74%|███████▍  | 7419/10000 [50:04<09:36,  4.47it/s]
 74%|███████▍  | 7420/10000 [50:04<09:53,  4.34it/s]
                                                    
{'loss': 0.0918, 'grad_norm': 0.620574951171875, 'learning_rate': 1.7133314417425594e-05}

 74%|███████▍  | 7420/10000 [50:04<09:53,  4.34it/s]
 74%|███████▍  | 7421/10000 [50:04<09:42,  4.43it/s]
 74%|███████▍  | 7422/10000 [50:05<09:22,  4.59it/s]
 74%|███████▍  | 7423/10000 [50:05<09:04,  4.74it/s]
 74%|███████▍  | 7424/10000 [50:05<08:51,  4.85it/s]
 74%|███████▍  | 7425/10000 [50:05<08:42,  4.93it/s]
 74%|███████▍  | 7426/10000 [50:05<08:59,  4.77it/s]
 74%|███████▍  | 7427/10000 [50:06<09:07,  4.70it/s]
 74%|███████▍  | 7428/10000 [50:06<09:07,  4.69it/s]
 74%|███████▍  | 7429/10000 [50:06<09:55,  4.32it/s]
 74%|███████▍  | 7430/10000 [50:06<09:37,  4.45it/s]
                                                    
{'loss': 0.0769, 'grad_norm': 0.6385549306869507, 'learning_rate': 1.7008888933844408e-05}

 74%|███████▍  | 7430/10000 [50:06<09:37,  4.45it/s]
 74%|███████▍  | 7431/10000 [50:07<09:28,  4.52it/s]
 74%|███████▍  | 7432/10000 [50:07<09:12,  4.65it/s]
 74%|███████▍  | 7433/10000 [50:07<08:49,  4.85it/s]
 74%|███████▍  | 7434/10000 [50:07<08:31,  5.02it/s]
 74%|███████▍  | 7435/10000 [50:07<08:19,  5.14it/s]
 74%|███████▍  | 7436/10000 [50:08<08:53,  4.81it/s]
 74%|███████▍  | 7437/10000 [50:08<09:20,  4.57it/s]
 74%|███████▍  | 7438/10000 [50:08<09:26,  4.53it/s]
 74%|███████▍  | 7439/10000 [50:08<09:28,  4.51it/s]
 74%|███████▍  | 7440/10000 [50:08<09:22,  4.55it/s]
                                                    
{'loss': 0.0676, 'grad_norm': 0.6161463856697083, 'learning_rate': 1.6884824235768172e-05}

 74%|███████▍  | 7440/10000 [50:09<09:22,  4.55it/s]
 74%|███████▍  | 7441/10000 [50:09<09:01,  4.72it/s]
 74%|███████▍  | 7442/10000 [50:09<08:38,  4.93it/s]
 74%|███████▍  | 7443/10000 [50:09<08:17,  5.14it/s]
 74%|███████▍  | 7444/10000 [50:09<08:23,  5.07it/s]
 74%|███████▍  | 7445/10000 [50:10<09:27,  4.50it/s]
 74%|███████▍  | 7446/10000 [50:10<09:25,  4.51it/s]
 74%|███████▍  | 7447/10000 [50:10<09:34,  4.44it/s]
 74%|███████▍  | 7448/10000 [50:10<09:37,  4.42it/s]
 74%|███████▍  | 7449/10000 [50:10<09:30,  4.47it/s]
 74%|███████▍  | 7450/10000 [50:11<09:08,  4.65it/s]
                                                    
{'loss': 0.0788, 'grad_norm': 0.5900981426239014, 'learning_rate': 1.6761121679948592e-05}

 74%|███████▍  | 7450/10000 [50:11<09:08,  4.65it/s]
 75%|███████▍  | 7451/10000 [50:11<08:56,  4.75it/s]
 75%|███████▍  | 7452/10000 [50:11<08:42,  4.87it/s]
 75%|███████▍  | 7453/10000 [50:11<08:46,  4.84it/s]
 75%|███████▍  | 7454/10000 [50:11<09:01,  4.70it/s]
 75%|███████▍  | 7455/10000 [50:12<09:16,  4.57it/s]
 75%|███████▍  | 7456/10000 [50:12<09:22,  4.52it/s]
 75%|███████▍  | 7457/10000 [50:12<09:11,  4.61it/s]
 75%|███████▍  | 7458/10000 [50:12<08:49,  4.80it/s]
 75%|███████▍  | 7459/10000 [50:13<08:40,  4.88it/s]
 75%|███████▍  | 7460/10000 [50:13<08:37,  4.90it/s]
                                                    
{'loss': 0.0802, 'grad_norm': 0.5868789553642273, 'learning_rate': 1.663778261917695e-05}

 75%|███████▍  | 7460/10000 [50:13<08:37,  4.90it/s]
 75%|███████▍  | 7461/10000 [50:13<09:01,  4.69it/s]
 75%|███████▍  | 7462/10000 [50:13<09:09,  4.62it/s]
 75%|███████▍  | 7463/10000 [50:13<09:12,  4.59it/s]
 75%|███████▍  | 7464/10000 [50:14<09:11,  4.60it/s]
 75%|███████▍  | 7465/10000 [50:14<08:56,  4.72it/s]
 75%|███████▍  | 7466/10000 [50:14<08:43,  4.84it/s]
 75%|███████▍  | 7467/10000 [50:14<08:49,  4.78it/s]
 75%|███████▍  | 7468/10000 [50:14<09:06,  4.63it/s]
 75%|███████▍  | 7469/10000 [50:15<09:23,  4.49it/s]
 75%|███████▍  | 7470/10000 [50:15<09:31,  4.42it/s]
                                                    
{'loss': 0.0818, 'grad_norm': 0.6996265053749084, 'learning_rate': 1.651480840226952e-05}

 75%|███████▍  | 7470/10000 [50:15<09:31,  4.42it/s]
 75%|███████▍  | 7471/10000 [50:15<09:38,  4.37it/s]
 75%|███████▍  | 7472/10000 [50:15<09:25,  4.47it/s]
 75%|███████▍  | 7473/10000 [50:16<09:12,  4.58it/s]
 75%|███████▍  | 7474/10000 [50:16<08:54,  4.73it/s]
 75%|███████▍  | 7475/10000 [50:16<08:39,  4.86it/s]
 75%|███████▍  | 7476/10000 [50:16<08:22,  5.02it/s]
 75%|███████▍  | 7477/10000 [50:16<08:35,  4.89it/s]
 75%|███████▍  | 7478/10000 [50:17<08:53,  4.73it/s]
 75%|███████▍  | 7479/10000 [50:17<09:06,  4.61it/s]
 75%|███████▍  | 7480/10000 [50:17<09:17,  4.52it/s]
                                                    
{'loss': 0.0773, 'grad_norm': 0.7780561447143555, 'learning_rate': 1.639220037405258e-05}

 75%|███████▍  | 7480/10000 [50:17<09:17,  4.52it/s]
 75%|███████▍  | 7481/10000 [50:17<09:16,  4.53it/s]
 75%|███████▍  | 7482/10000 [50:17<08:55,  4.70it/s]
 75%|███████▍  | 7483/10000 [50:18<08:41,  4.83it/s]
 75%|███████▍  | 7484/10000 [50:18<08:30,  4.93it/s]
 75%|███████▍  | 7485/10000 [50:18<08:28,  4.95it/s]
 75%|███████▍  | 7486/10000 [50:18<08:44,  4.79it/s]
 75%|███████▍  | 7487/10000 [50:18<08:59,  4.66it/s]
 75%|███████▍  | 7488/10000 [50:19<08:56,  4.68it/s]
 75%|███████▍  | 7489/10000 [50:19<09:01,  4.64it/s]
 75%|███████▍  | 7490/10000 [50:19<09:03,  4.62it/s]
                                                    
{'loss': 0.0813, 'grad_norm': 0.6281243562698364, 'learning_rate': 1.6269959875347906e-05}

 75%|███████▍  | 7490/10000 [50:19<09:03,  4.62it/s]
 75%|███████▍  | 7491/10000 [50:19<09:01,  4.63it/s]
 75%|███████▍  | 7492/10000 [50:20<08:37,  4.85it/s]
 75%|███████▍  | 7493/10000 [50:20<08:38,  4.83it/s]
 75%|███████▍  | 7494/10000 [50:20<08:50,  4.72it/s]
 75%|███████▍  | 7495/10000 [50:20<09:04,  4.60it/s]
 75%|███████▍  | 7496/10000 [50:20<09:26,  4.42it/s]
 75%|███████▍  | 7497/10000 [50:21<09:33,  4.36it/s]
 75%|███████▍  | 7498/10000 [50:21<09:34,  4.36it/s]
 75%|███████▍  | 7499/10000 [50:21<09:36,  4.34it/s]
 75%|███████▌  | 7500/10000 [50:21<09:18,  4.48it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.73 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.74 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.06 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.13 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.13 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.36 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 7500: eval/loss = 0.0937 (36s)

                                                    
{'loss': 0.06, 'grad_norm': 0.8893796801567078, 'learning_rate': 1.614808824295802e-05}

 75%|███████▌  | 7500/10000 [50:58<09:18,  4.48it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-7500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-7500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-7500/wandb_config.json

 75%|███████▌  | 7501/10000 [51:26<13:29:26, 19.43s/it]
 75%|███████▌  | 7502/10000 [51:26<9:28:55, 13.67s/it] 
 75%|███████▌  | 7503/10000 [51:26<6:40:48,  9.63s/it]
 75%|███████▌  | 7504/10000 [51:26<4:43:05,  6.80s/it]
 75%|███████▌  | 7505/10000 [51:26<3:20:48,  4.83s/it]
 75%|███████▌  | 7506/10000 [51:27<2:22:53,  3.44s/it]
 75%|███████▌  | 7507/10000 [51:27<1:42:19,  2.46s/it]
 75%|███████▌  | 7508/10000 [51:27<1:14:00,  1.78s/it]
 75%|███████▌  | 7509/10000 [51:27<54:01,  1.30s/it]  
 75%|███████▌  | 7510/10000 [51:27<40:02,  1.04it/s]
                                                    
{'loss': 0.0742, 'grad_norm': 0.6010493636131287, 'learning_rate': 1.602658680965152e-05}

 75%|███████▌  | 7510/10000 [51:27<40:02,  1.04it/s]
 75%|███████▌  | 7511/10000 [51:28<30:33,  1.36it/s]
 75%|███████▌  | 7512/10000 [51:28<23:50,  1.74it/s]
 75%|███████▌  | 7513/10000 [51:28<19:18,  2.15it/s]
 75%|███████▌  | 7514/10000 [51:28<16:26,  2.52it/s]
 75%|███████▌  | 7515/10000 [51:28<13:56,  2.97it/s]
 75%|███████▌  | 7516/10000 [51:29<12:19,  3.36it/s]
 75%|███████▌  | 7517/10000 [51:29<11:18,  3.66it/s]
 75%|███████▌  | 7518/10000 [51:29<10:40,  3.87it/s]
 75%|███████▌  | 7519/10000 [51:29<10:29,  3.94it/s]
 75%|███████▌  | 7520/10000 [51:30<10:14,  4.04it/s]
                                                    
{'loss': 0.0935, 'grad_norm': 0.6025218963623047, 'learning_rate': 1.5905456904148686e-05}

 75%|███████▌  | 7520/10000 [51:30<10:14,  4.04it/s]
 75%|███████▌  | 7521/10000 [51:30<10:19,  4.00it/s]
 75%|███████▌  | 7522/10000 [51:30<09:39,  4.28it/s]
 75%|███████▌  | 7523/10000 [51:30<09:16,  4.45it/s]
 75%|███████▌  | 7524/10000 [51:30<08:58,  4.60it/s]
 75%|███████▌  | 7525/10000 [51:31<08:46,  4.70it/s]
 75%|███████▌  | 7526/10000 [51:31<08:39,  4.76it/s]
 75%|███████▌  | 7527/10000 [51:31<08:39,  4.76it/s]
 75%|███████▌  | 7528/10000 [51:31<08:54,  4.63it/s]
 75%|███████▌  | 7529/10000 [51:31<08:51,  4.65it/s]
 75%|███████▌  | 7530/10000 [51:32<08:48,  4.68it/s]
                                                    
{'loss': 0.0854, 'grad_norm': 0.8695430755615234, 'learning_rate': 1.57846998511067e-05}

 75%|███████▌  | 7530/10000 [51:32<08:48,  4.68it/s]
 75%|███████▌  | 7531/10000 [51:32<08:48,  4.67it/s]
 75%|███████▌  | 7532/10000 [51:32<08:33,  4.81it/s]
 75%|███████▌  | 7533/10000 [51:32<08:24,  4.89it/s]
 75%|███████▌  | 7534/10000 [51:32<08:08,  5.04it/s]
 75%|███████▌  | 7535/10000 [51:33<07:58,  5.15it/s]
 75%|███████▌  | 7536/10000 [51:33<07:53,  5.21it/s]
 75%|███████▌  | 7537/10000 [51:33<08:04,  5.08it/s]
 75%|███████▌  | 7538/10000 [51:33<08:24,  4.88it/s]
 75%|███████▌  | 7539/10000 [51:34<08:37,  4.76it/s]
 75%|███████▌  | 7540/10000 [51:34<08:41,  4.72it/s]
                                                    
{'loss': 0.087, 'grad_norm': 0.8973163366317749, 'learning_rate': 1.566431697110538e-05}

 75%|███████▌  | 7540/10000 [51:34<08:41,  4.72it/s]
 75%|███████▌  | 7541/10000 [51:34<08:57,  4.57it/s]
 75%|███████▌  | 7542/10000 [51:34<08:43,  4.69it/s]
 75%|███████▌  | 7543/10000 [51:34<08:28,  4.83it/s]
 75%|███████▌  | 7544/10000 [51:35<08:07,  5.04it/s]
 75%|███████▌  | 7545/10000 [51:35<07:59,  5.12it/s]
 75%|███████▌  | 7546/10000 [51:35<08:05,  5.06it/s]Rank 0, Worker 4: Wait for shard 27 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 75%|███████▌  | 7547/10000 [51:35<08:20,  4.90it/s]
 75%|███████▌  | 7548/10000 [51:35<08:48,  4.64it/s]
 75%|███████▌  | 7549/10000 [51:36<08:58,  4.55it/s]
 76%|███████▌  | 7550/10000 [51:36<09:05,  4.49it/s]
                                                    
{'loss': 0.0746, 'grad_norm': 0.6022163033485413, 'learning_rate': 1.554430958063259e-05}

 76%|███████▌  | 7550/10000 [51:36<09:05,  4.49it/s]Rank 0, Worker 2: Wait for shard 17 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 76%|███████▌  | 7551/10000 [51:36<09:10,  4.45it/s]
 76%|███████▌  | 7552/10000 [51:36<08:52,  4.60it/s]
 76%|███████▌  | 7553/10000 [51:36<08:36,  4.74it/s]
 76%|███████▌  | 7554/10000 [51:37<08:34,  4.75it/s]
 76%|███████▌  | 7555/10000 [51:37<08:50,  4.61it/s]
 76%|███████▌  | 7556/10000 [51:37<09:05,  4.48it/s]
 76%|███████▌  | 7557/10000 [51:37<09:22,  4.34it/s]
 76%|███████▌  | 7558/10000 [51:38<09:14,  4.40it/s]
 76%|███████▌  | 7559/10000 [51:38<09:07,  4.46it/s]
 76%|███████▌  | 7560/10000 [51:38<08:51,  4.59it/s]
                                                    
{'loss': 0.0688, 'grad_norm': 0.9059644341468811, 'learning_rate': 1.5424678992069912e-05}

 76%|███████▌  | 7560/10000 [51:38<08:51,  4.59it/s]
 76%|███████▌  | 7561/10000 [51:38<09:15,  4.39it/s]
 76%|███████▌  | 7562/10000 [51:39<09:29,  4.28it/s]
 76%|███████▌  | 7563/10000 [51:39<09:31,  4.26it/s]
 76%|███████▌  | 7564/10000 [51:39<09:51,  4.12it/s]
 76%|███████▌  | 7565/10000 [51:39<09:49,  4.13it/s]
 76%|███████▌  | 7566/10000 [51:40<09:46,  4.15it/s]
 76%|███████▌  | 7567/10000 [51:40<09:12,  4.40it/s]Rank 0, Worker 1: Wait for shard 55 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 76%|███████▌  | 7568/10000 [51:40<09:01,  4.49it/s]
 76%|███████▌  | 7569/10000 [51:40<08:48,  4.60it/s]
 76%|███████▌  | 7570/10000 [51:40<08:43,  4.64it/s]
                                                    
{'loss': 0.0603, 'grad_norm': 0.7430972456932068, 'learning_rate': 1.5305426513678362e-05}

 76%|███████▌  | 7570/10000 [51:40<08:43,  4.64it/s]
 76%|███████▌  | 7571/10000 [51:41<09:01,  4.48it/s]
 76%|███████▌  | 7572/10000 [51:41<09:08,  4.43it/s]
 76%|███████▌  | 7573/10000 [51:41<09:20,  4.33it/s]
 76%|███████▌  | 7574/10000 [51:41<09:15,  4.37it/s]
 76%|███████▌  | 7575/10000 [51:41<09:04,  4.45it/s]
 76%|███████▌  | 7576/10000 [51:42<08:51,  4.56it/s]
 76%|███████▌  | 7577/10000 [51:42<08:46,  4.60it/s]
 76%|███████▌  | 7578/10000 [51:42<08:54,  4.53it/s]
 76%|███████▌  | 7579/10000 [51:42<09:05,  4.44it/s]
 76%|███████▌  | 7580/10000 [51:43<09:12,  4.38it/s]
                                                    
{'loss': 0.0842, 'grad_norm': 0.5842188596725464, 'learning_rate': 1.518655344958388e-05}

 76%|███████▌  | 7580/10000 [51:43<09:12,  4.38it/s]
 76%|███████▌  | 7581/10000 [51:43<09:43,  4.15it/s]
 76%|███████▌  | 7582/10000 [51:43<09:22,  4.30it/s]
 76%|███████▌  | 7583/10000 [51:43<09:00,  4.47it/s]
 76%|███████▌  | 7584/10000 [51:43<08:47,  4.58it/s]
 76%|███████▌  | 7585/10000 [51:44<08:55,  4.51it/s]
 76%|███████▌  | 7586/10000 [51:44<08:59,  4.47it/s]
 76%|███████▌  | 7587/10000 [51:44<09:05,  4.42it/s]
 76%|███████▌  | 7588/10000 [51:44<09:19,  4.31it/s]
 76%|███████▌  | 7589/10000 [51:45<09:32,  4.21it/s]
 76%|███████▌  | 7590/10000 [51:45<09:43,  4.13it/s]
                                                    
{'loss': 0.0738, 'grad_norm': 0.636116087436676, 'learning_rate': 1.5068061099763275e-05}

 76%|███████▌  | 7590/10000 [51:45<09:43,  4.13it/s]
 76%|███████▌  | 7591/10000 [51:45<09:45,  4.11it/s]
 76%|███████▌  | 7592/10000 [51:45<09:52,  4.06it/s]
 76%|███████▌  | 7593/10000 [51:46<09:50,  4.08it/s]
 76%|███████▌  | 7594/10000 [51:46<09:37,  4.17it/s]
 76%|███████▌  | 7595/10000 [51:46<09:55,  4.04it/s]
 76%|███████▌  | 7596/10000 [51:46<10:01,  4.00it/s]
 76%|███████▌  | 7597/10000 [51:47<10:18,  3.88it/s]
 76%|███████▌  | 7598/10000 [51:47<10:17,  3.89it/s]
 76%|███████▌  | 7599/10000 [51:47<11:06,  3.60it/s]
 76%|███████▌  | 7600/10000 [51:48<10:41,  3.74it/s]
                                                    
{'loss': 0.0858, 'grad_norm': 0.6917994022369385, 'learning_rate': 1.494995076002988e-05}

 76%|███████▌  | 7600/10000 [51:48<10:41,  3.74it/s]
 76%|███████▌  | 7601/10000 [51:48<10:25,  3.83it/s]
 76%|███████▌  | 7602/10000 [51:48<10:08,  3.94it/s]
 76%|███████▌  | 7603/10000 [51:48<09:36,  4.16it/s]
 76%|███████▌  | 7604/10000 [51:48<09:20,  4.28it/s]
 76%|███████▌  | 7605/10000 [51:49<09:20,  4.28it/s]
 76%|███████▌  | 7606/10000 [51:49<08:59,  4.44it/s]
 76%|███████▌  | 7607/10000 [51:49<09:00,  4.43it/s]
 76%|███████▌  | 7608/10000 [51:49<08:59,  4.43it/s]
 76%|███████▌  | 7609/10000 [51:50<09:00,  4.43it/s]
 76%|███████▌  | 7610/10000 [51:50<09:11,  4.34it/s]
                                                    
{'loss': 0.0727, 'grad_norm': 0.8647133111953735, 'learning_rate': 1.4832223722019456e-05}

 76%|███████▌  | 7610/10000 [51:50<09:11,  4.34it/s]
 76%|███████▌  | 7611/10000 [51:50<09:11,  4.33it/s]
 76%|███████▌  | 7612/10000 [51:50<08:54,  4.47it/s]
 76%|███████▌  | 7613/10000 [51:50<08:38,  4.61it/s]
 76%|███████▌  | 7614/10000 [51:51<08:43,  4.56it/s]
 76%|███████▌  | 7615/10000 [51:51<08:49,  4.50it/s]
 76%|███████▌  | 7616/10000 [51:51<08:57,  4.43it/s]
 76%|███████▌  | 7617/10000 [51:51<09:07,  4.35it/s]
 76%|███████▌  | 7618/10000 [51:52<08:59,  4.41it/s]
 76%|███████▌  | 7619/10000 [51:52<08:50,  4.49it/s]
 76%|███████▌  | 7620/10000 [51:52<08:49,  4.49it/s]
                                                    
{'loss': 0.0886, 'grad_norm': 1.0176072120666504, 'learning_rate': 1.4714881273176035e-05}

 76%|███████▌  | 7620/10000 [51:52<08:49,  4.49it/s]
 76%|███████▌  | 7621/10000 [51:52<09:52,  4.02it/s]
 76%|███████▌  | 7622/10000 [51:53<09:22,  4.23it/s]
 76%|███████▌  | 7623/10000 [51:53<09:20,  4.24it/s]
 76%|███████▌  | 7624/10000 [51:53<09:53,  4.00it/s]
 76%|███████▋  | 7625/10000 [51:53<09:52,  4.01it/s]
 76%|███████▋  | 7626/10000 [51:54<09:52,  4.01it/s]
 76%|███████▋  | 7627/10000 [51:54<09:48,  4.03it/s]
 76%|███████▋  | 7628/10000 [51:54<09:27,  4.18it/s]
 76%|███████▋  | 7629/10000 [51:54<09:20,  4.23it/s]
 76%|███████▋  | 7630/10000 [51:54<09:07,  4.33it/s]
                                                    
{'loss': 0.0628, 'grad_norm': 0.716916561126709, 'learning_rate': 1.4597924696737835e-05}

 76%|███████▋  | 7630/10000 [51:55<09:07,  4.33it/s]
 76%|███████▋  | 7631/10000 [51:55<09:31,  4.14it/s]
 76%|███████▋  | 7632/10000 [51:55<09:04,  4.35it/s]
 76%|███████▋  | 7633/10000 [51:55<09:12,  4.29it/s]
 76%|███████▋  | 7634/10000 [51:55<09:18,  4.24it/s]
 76%|███████▋  | 7635/10000 [51:56<09:22,  4.20it/s]
 76%|███████▋  | 7636/10000 [51:56<09:13,  4.27it/s]
 76%|███████▋  | 7637/10000 [51:56<08:54,  4.42it/s]
 76%|███████▋  | 7638/10000 [51:56<08:37,  4.57it/s]
 76%|███████▋  | 7639/10000 [51:57<09:06,  4.32it/s]
 76%|███████▋  | 7640/10000 [51:57<08:52,  4.44it/s]
                                                    
{'loss': 0.0742, 'grad_norm': 0.6679605841636658, 'learning_rate': 1.4481355271723252e-05}

 76%|███████▋  | 7640/10000 [51:57<08:52,  4.44it/s]
 76%|███████▋  | 7641/10000 [51:57<08:48,  4.46it/s]
 76%|███████▋  | 7642/10000 [51:57<08:52,  4.43it/s]
 76%|███████▋  | 7643/10000 [51:57<08:57,  4.38it/s]
 76%|███████▋  | 7644/10000 [51:58<09:01,  4.35it/s]
 76%|███████▋  | 7645/10000 [51:58<08:52,  4.42it/s]
 76%|███████▋  | 7646/10000 [51:58<08:29,  4.62it/s]
 76%|███████▋  | 7647/10000 [51:58<08:07,  4.82it/s]
 76%|███████▋  | 7648/10000 [51:58<08:04,  4.86it/s]
 76%|███████▋  | 7649/10000 [51:59<08:11,  4.78it/s]
 76%|███████▋  | 7650/10000 [51:59<08:20,  4.70it/s]
                                                    
{'loss': 0.0818, 'grad_norm': 0.7037312388420105, 'learning_rate': 1.4365174272916809e-05}

 76%|███████▋  | 7650/10000 [51:59<08:20,  4.70it/s]
 77%|███████▋  | 7651/10000 [51:59<08:37,  4.54it/s]
 77%|███████▋  | 7652/10000 [51:59<08:43,  4.49it/s]
 77%|███████▋  | 7653/10000 [52:00<08:40,  4.51it/s]
 77%|███████▋  | 7654/10000 [52:00<08:40,  4.51it/s]
 77%|███████▋  | 7655/10000 [52:00<08:26,  4.63it/s]
 77%|███████▋  | 7656/10000 [52:00<08:15,  4.73it/s]
 77%|███████▋  | 7657/10000 [52:00<08:11,  4.77it/s]
 77%|███████▋  | 7658/10000 [52:01<08:25,  4.63it/s]
 77%|███████▋  | 7659/10000 [52:01<08:24,  4.64it/s]
 77%|███████▋  | 7660/10000 [52:01<08:32,  4.56it/s]
                                                    
{'loss': 0.0639, 'grad_norm': 0.5630443096160889, 'learning_rate': 1.4249382970855319e-05}

 77%|███████▋  | 7660/10000 [52:01<08:32,  4.56it/s]
 77%|███████▋  | 7661/10000 [52:01<08:35,  4.54it/s]
 77%|███████▋  | 7662/10000 [52:02<08:25,  4.62it/s]
 77%|███████▋  | 7663/10000 [52:02<08:17,  4.69it/s]
 77%|███████▋  | 7664/10000 [52:02<08:17,  4.70it/s]
 77%|███████▋  | 7665/10000 [52:02<08:18,  4.68it/s]
 77%|███████▋  | 7666/10000 [52:02<08:22,  4.64it/s]
 77%|███████▋  | 7667/10000 [52:03<08:17,  4.69it/s]
 77%|███████▋  | 7668/10000 [52:03<08:14,  4.71it/s]
 77%|███████▋  | 7669/10000 [52:03<08:09,  4.76it/s]
 77%|███████▋  | 7670/10000 [52:03<07:50,  4.96it/s]
                                                    
{'loss': 0.0747, 'grad_norm': 0.5449936389923096, 'learning_rate': 1.4133982631813903e-05}

 77%|███████▋  | 7670/10000 [52:03<07:50,  4.96it/s]
 77%|███████▋  | 7671/10000 [52:03<07:59,  4.86it/s]
 77%|███████▋  | 7672/10000 [52:04<07:53,  4.91it/s]
 77%|███████▋  | 7673/10000 [52:04<07:44,  5.01it/s]
 77%|███████▋  | 7674/10000 [52:04<07:46,  4.98it/s]
 77%|███████▋  | 7675/10000 [52:04<07:42,  5.02it/s]
 77%|███████▋  | 7676/10000 [52:04<07:33,  5.13it/s]
 77%|███████▋  | 7677/10000 [52:05<07:24,  5.23it/s]
 77%|███████▋  | 7678/10000 [52:05<07:15,  5.33it/s]
 77%|███████▋  | 7679/10000 [52:05<07:07,  5.43it/s]
 77%|███████▋  | 7680/10000 [52:05<07:05,  5.45it/s]
                                                    
{'loss': 0.0652, 'grad_norm': 0.796459972858429, 'learning_rate': 1.4018974517792194e-05}

 77%|███████▋  | 7680/10000 [52:05<07:05,  5.45it/s]
 77%|███████▋  | 7681/10000 [52:05<07:13,  5.35it/s]
 77%|███████▋  | 7682/10000 [52:05<07:15,  5.32it/s]
 77%|███████▋  | 7683/10000 [52:06<07:19,  5.27it/s]
 77%|███████▋  | 7684/10000 [52:06<07:20,  5.26it/s]
 77%|███████▋  | 7685/10000 [52:06<07:13,  5.34it/s]
 77%|███████▋  | 7686/10000 [52:06<07:09,  5.38it/s]
 77%|███████▋  | 7687/10000 [52:06<07:01,  5.48it/s]
 77%|███████▋  | 7688/10000 [52:07<06:57,  5.53it/s]
 77%|███████▋  | 7689/10000 [52:07<06:53,  5.58it/s]
 77%|███████▋  | 7690/10000 [52:07<06:59,  5.50it/s]
                                                    
{'loss': 0.0553, 'grad_norm': 1.1005202531814575, 'learning_rate': 1.390435988650048e-05}

 77%|███████▋  | 7690/10000 [52:07<06:59,  5.50it/s]
 77%|███████▋  | 7691/10000 [52:07<07:02,  5.46it/s]
 77%|███████▋  | 7692/10000 [52:07<06:59,  5.51it/s]
 77%|███████▋  | 7693/10000 [52:07<06:55,  5.56it/s]
 77%|███████▋  | 7694/10000 [52:08<06:55,  5.55it/s]
 77%|███████▋  | 7695/10000 [52:08<06:55,  5.55it/s]Rank 0, Worker 3: Wait for shard 64 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 77%|███████▋  | 7696/10000 [52:08<06:56,  5.53it/s]
 77%|███████▋  | 7697/10000 [52:08<06:59,  5.48it/s]
 77%|███████▋  | 7698/10000 [52:08<07:03,  5.44it/s]
 77%|███████▋  | 7699/10000 [52:09<07:06,  5.39it/s]
 77%|███████▋  | 7700/10000 [52:09<07:02,  5.44it/s]
                                                    
{'loss': 0.0718, 'grad_norm': 0.780353844165802, 'learning_rate': 1.3790139991346006e-05}

 77%|███████▋  | 7700/10000 [52:09<07:02,  5.44it/s]
 77%|███████▋  | 7701/10000 [52:09<07:02,  5.44it/s]
 77%|███████▋  | 7702/10000 [52:09<06:56,  5.52it/s]
 77%|███████▋  | 7703/10000 [52:09<06:51,  5.58it/s]
 77%|███████▋  | 7704/10000 [52:09<06:52,  5.57it/s]
 77%|███████▋  | 7705/10000 [52:10<07:00,  5.46it/s]
 77%|███████▋  | 7706/10000 [52:10<07:05,  5.39it/s]
 77%|███████▋  | 7707/10000 [52:10<07:09,  5.34it/s]
 77%|███████▋  | 7708/10000 [52:10<07:08,  5.35it/s]
 77%|███████▋  | 7709/10000 [52:10<06:57,  5.49it/s]
 77%|███████▋  | 7710/10000 [52:11<06:52,  5.56it/s]
                                                    
{'loss': 0.1103, 'grad_norm': 0.7461047172546387, 'learning_rate': 1.367631608141926e-05}

 77%|███████▋  | 7710/10000 [52:11<06:52,  5.56it/s]
 77%|███████▋  | 7711/10000 [52:11<06:54,  5.52it/s]
 77%|███████▋  | 7712/10000 [52:11<06:49,  5.59it/s]
 77%|███████▋  | 7713/10000 [52:11<06:52,  5.54it/s]
 77%|███████▋  | 7714/10000 [52:11<07:04,  5.38it/s]
 77%|███████▋  | 7715/10000 [52:12<07:10,  5.31it/s]
 77%|███████▋  | 7716/10000 [52:12<07:17,  5.22it/s]
 77%|███████▋  | 7717/10000 [52:12<07:17,  5.22it/s]
 77%|███████▋  | 7718/10000 [52:12<07:07,  5.34it/s]
 77%|███████▋  | 7719/10000 [52:12<06:59,  5.44it/s]
 77%|███████▋  | 7720/10000 [52:12<06:53,  5.52it/s]
                                                    
{'loss': 0.0672, 'grad_norm': 0.8094848990440369, 'learning_rate': 1.3562889401480278e-05}

 77%|███████▋  | 7720/10000 [52:12<06:53,  5.52it/s]
 77%|███████▋  | 7721/10000 [52:13<06:56,  5.47it/s]
 77%|███████▋  | 7722/10000 [52:13<07:00,  5.42it/s]
 77%|███████▋  | 7723/10000 [52:13<07:08,  5.31it/s]
 77%|███████▋  | 7724/10000 [52:13<07:12,  5.26it/s]
 77%|███████▋  | 7725/10000 [52:13<07:13,  5.24it/s]
 77%|███████▋  | 7726/10000 [52:14<07:09,  5.30it/s]
 77%|███████▋  | 7727/10000 [52:14<06:57,  5.44it/s]
 77%|███████▋  | 7728/10000 [52:14<06:51,  5.52it/s]
 77%|███████▋  | 7729/10000 [52:14<06:51,  5.52it/s]
 77%|███████▋  | 7730/10000 [52:14<06:59,  5.41it/s]
                                                    
{'loss': 0.0696, 'grad_norm': 0.544795036315918, 'learning_rate': 1.3449861191945074e-05}

 77%|███████▋  | 7730/10000 [52:14<06:59,  5.41it/s]
 77%|███████▋  | 7731/10000 [52:15<07:19,  5.16it/s]
 77%|███████▋  | 7732/10000 [52:15<07:25,  5.09it/s]
 77%|███████▋  | 7733/10000 [52:15<07:26,  5.07it/s]
 77%|███████▋  | 7734/10000 [52:15<07:16,  5.19it/s]
 77%|███████▋  | 7735/10000 [52:15<07:05,  5.32it/s]
 77%|███████▋  | 7736/10000 [52:15<06:56,  5.44it/s]
 77%|███████▋  | 7737/10000 [52:16<07:09,  5.27it/s]
 77%|███████▋  | 7738/10000 [52:16<07:30,  5.02it/s]
 77%|███████▋  | 7739/10000 [52:16<07:49,  4.81it/s]Rank 0, Worker 5: Wait for shard 1 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 77%|███████▋  | 7740/10000 [52:16<07:46,  4.84it/s]
                                                    
{'loss': 0.0772, 'grad_norm': 0.6924198269844055, 'learning_rate': 1.3337232688872009e-05}

 77%|███████▋  | 7740/10000 [52:16<07:46,  4.84it/s]Rank 0, Worker 0: Wait for shard 59 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 77%|███████▋  | 7741/10000 [52:17<07:55,  4.75it/s]
 77%|███████▋  | 7742/10000 [52:17<07:51,  4.79it/s]
 77%|███████▋  | 7743/10000 [52:17<07:43,  4.87it/s]
 77%|███████▋  | 7744/10000 [52:17<07:36,  4.94it/s]
 77%|███████▋  | 7745/10000 [52:17<07:27,  5.04it/s]
 77%|███████▋  | 7746/10000 [52:18<07:19,  5.13it/s]
 77%|███████▋  | 7747/10000 [52:18<07:16,  5.16it/s]
 77%|███████▋  | 7748/10000 [52:18<07:16,  5.16it/s]
 77%|███████▋  | 7749/10000 [52:18<07:26,  5.05it/s]
 78%|███████▊  | 7750/10000 [52:18<07:45,  4.84it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Caching shard...Rank 0, Worker 1: Caching shard...

Rank 0, Worker 4: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.61 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.07 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 18.14 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 18.17 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 18.20 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 18.28 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 7750: eval/loss = 0.0895 (36s)

                                                    
{'loss': 0.0727, 'grad_norm': 0.6202091574668884, 'learning_rate': 1.3225005123948364e-05}

 78%|███████▊  | 7750/10000 [52:55<07:45,  4.84it/s]
 78%|███████▊  | 7751/10000 [52:55<6:56:12, 11.10s/it]
 78%|███████▊  | 7752/10000 [52:55<4:53:12,  7.83s/it]
 78%|███████▊  | 7753/10000 [52:55<3:27:15,  5.53s/it]
 78%|███████▊  | 7754/10000 [52:55<2:27:11,  3.93s/it]
 78%|███████▊  | 7755/10000 [52:56<1:45:10,  2.81s/it]
 78%|███████▊  | 7756/10000 [52:56<1:15:48,  2.03s/it]
 78%|███████▊  | 7757/10000 [52:56<55:28,  1.48s/it]  
 78%|███████▊  | 7758/10000 [52:56<41:11,  1.10s/it]
 78%|███████▊  | 7759/10000 [52:56<30:59,  1.21it/s]
 78%|███████▊  | 7760/10000 [52:57<24:05,  1.55it/s]
                                                    
{'loss': 0.0863, 'grad_norm': 0.6977066397666931, 'learning_rate': 1.311317972447681e-05}

 78%|███████▊  | 7760/10000 [52:57<24:05,  1.55it/s]
 78%|███████▊  | 7761/10000 [52:57<19:23,  1.92it/s]
 78%|███████▊  | 7762/10000 [52:57<16:04,  2.32it/s]
 78%|███████▊  | 7763/10000 [52:57<13:37,  2.74it/s]
 78%|███████▊  | 7764/10000 [52:58<11:44,  3.17it/s]
 78%|███████▊  | 7765/10000 [52:58<10:36,  3.51it/s]
 78%|███████▊  | 7766/10000 [52:58<09:35,  3.88it/s]
 78%|███████▊  | 7767/10000 [52:58<08:52,  4.19it/s]
 78%|███████▊  | 7768/10000 [52:58<08:25,  4.42it/s]
 78%|███████▊  | 7769/10000 [52:59<08:05,  4.59it/s]
 78%|███████▊  | 7770/10000 [52:59<08:13,  4.52it/s]
                                                    
{'loss': 0.0818, 'grad_norm': 0.6502335667610168, 'learning_rate': 1.3001757713361996e-05}

 78%|███████▊  | 7770/10000 [52:59<08:13,  4.52it/s]
 78%|███████▊  | 7771/10000 [52:59<08:15,  4.50it/s]
 78%|███████▊  | 7772/10000 [52:59<08:06,  4.58it/s]
 78%|███████▊  | 7773/10000 [52:59<08:03,  4.61it/s]
 78%|███████▊  | 7774/10000 [53:00<07:35,  4.89it/s]
 78%|███████▊  | 7775/10000 [53:00<07:19,  5.06it/s]
 78%|███████▊  | 7776/10000 [53:00<07:18,  5.07it/s]
 78%|███████▊  | 7777/10000 [53:00<07:04,  5.24it/s]
 78%|███████▊  | 7778/10000 [53:00<07:02,  5.26it/s]
 78%|███████▊  | 7779/10000 [53:01<07:08,  5.18it/s]
 78%|███████▊  | 7780/10000 [53:01<07:09,  5.16it/s]
                                                    
{'loss': 0.0635, 'grad_norm': 0.8052932024002075, 'learning_rate': 1.2890740309097204e-05}

 78%|███████▊  | 7780/10000 [53:01<07:09,  5.16it/s]
 78%|███████▊  | 7781/10000 [53:01<07:17,  5.07it/s]
 78%|███████▊  | 7782/10000 [53:01<07:13,  5.12it/s]
 78%|███████▊  | 7783/10000 [53:01<07:03,  5.23it/s]
 78%|███████▊  | 7784/10000 [53:01<06:54,  5.35it/s]
 78%|███████▊  | 7785/10000 [53:02<06:48,  5.43it/s]
 78%|███████▊  | 7786/10000 [53:02<06:42,  5.51it/s]
 78%|███████▊  | 7787/10000 [53:02<06:43,  5.48it/s]
 78%|███████▊  | 7788/10000 [53:02<06:51,  5.37it/s]
 78%|███████▊  | 7789/10000 [53:02<07:05,  5.20it/s]
 78%|███████▊  | 7790/10000 [53:03<07:12,  5.11it/s]
                                                    
{'loss': 0.0785, 'grad_norm': 0.6477410793304443, 'learning_rate': 1.2780128725750944e-05}

 78%|███████▊  | 7790/10000 [53:03<07:12,  5.11it/s]
 78%|███████▊  | 7791/10000 [53:03<07:15,  5.07it/s]
 78%|███████▊  | 7792/10000 [53:03<07:00,  5.25it/s]
 78%|███████▊  | 7793/10000 [53:03<06:53,  5.34it/s]
 78%|███████▊  | 7794/10000 [53:03<06:48,  5.41it/s]
 78%|███████▊  | 7795/10000 [53:04<06:45,  5.44it/s]
 78%|███████▊  | 7796/10000 [53:04<06:58,  5.27it/s]
 78%|███████▊  | 7797/10000 [53:04<07:16,  5.05it/s]
 78%|███████▊  | 7798/10000 [53:04<07:06,  5.16it/s]
 78%|███████▊  | 7799/10000 [53:04<07:03,  5.19it/s]
 78%|███████▊  | 7800/10000 [53:04<06:52,  5.34it/s]
                                                    
{'loss': 0.0639, 'grad_norm': 0.5985831022262573, 'learning_rate': 1.266992417295379e-05}

 78%|███████▊  | 7800/10000 [53:05<06:52,  5.34it/s]
 78%|███████▊  | 7801/10000 [53:05<06:53,  5.32it/s]
 78%|███████▊  | 7802/10000 [53:05<06:46,  5.41it/s]
 78%|███████▊  | 7803/10000 [53:05<06:47,  5.39it/s]
 78%|███████▊  | 7804/10000 [53:05<06:56,  5.28it/s]
 78%|███████▊  | 7805/10000 [53:05<06:54,  5.30it/s]
 78%|███████▊  | 7806/10000 [53:06<06:52,  5.31it/s]
 78%|███████▊  | 7807/10000 [53:06<06:49,  5.36it/s]
 78%|███████▊  | 7808/10000 [53:06<06:43,  5.43it/s]
 78%|███████▊  | 7809/10000 [53:06<06:47,  5.37it/s]
 78%|███████▊  | 7810/10000 [53:06<06:54,  5.29it/s]
                                                    
{'loss': 0.0666, 'grad_norm': 0.7635859251022339, 'learning_rate': 1.2560127855885073e-05}

 78%|███████▊  | 7810/10000 [53:06<06:54,  5.29it/s]
 78%|███████▊  | 7811/10000 [53:07<07:03,  5.16it/s]
 78%|███████▊  | 7812/10000 [53:07<07:01,  5.19it/s]
 78%|███████▊  | 7813/10000 [53:07<06:59,  5.22it/s]
 78%|███████▊  | 7814/10000 [53:07<06:53,  5.28it/s]
 78%|███████▊  | 7815/10000 [53:07<06:48,  5.35it/s]
 78%|███████▊  | 7816/10000 [53:08<06:52,  5.29it/s]
 78%|███████▊  | 7817/10000 [53:08<06:58,  5.22it/s]
 78%|███████▊  | 7818/10000 [53:08<07:02,  5.17it/s]
 78%|███████▊  | 7819/10000 [53:08<07:07,  5.11it/s]
 78%|███████▊  | 7820/10000 [53:08<07:10,  5.06it/s]
                                                    
{'loss': 0.0848, 'grad_norm': 0.6572693586349487, 'learning_rate': 1.2450740975259745e-05}

 78%|███████▊  | 7820/10000 [53:08<07:10,  5.06it/s]
 78%|███████▊  | 7821/10000 [53:09<07:10,  5.06it/s]
 78%|███████▊  | 7822/10000 [53:09<07:04,  5.13it/s]
 78%|███████▊  | 7823/10000 [53:09<06:57,  5.22it/s]
 78%|███████▊  | 7824/10000 [53:09<06:58,  5.20it/s]
 78%|███████▊  | 7825/10000 [53:09<06:56,  5.22it/s]
 78%|███████▊  | 7826/10000 [53:09<06:59,  5.19it/s]
 78%|███████▊  | 7827/10000 [53:10<06:50,  5.29it/s]
 78%|███████▊  | 7828/10000 [53:10<06:44,  5.37it/s]
 78%|███████▊  | 7829/10000 [53:10<06:42,  5.39it/s]
 78%|███████▊  | 7830/10000 [53:10<06:41,  5.41it/s]
                                                    
{'loss': 0.0655, 'grad_norm': 0.7293024063110352, 'learning_rate': 1.234176472731517e-05}

 78%|███████▊  | 7830/10000 [53:10<06:41,  5.41it/s]
 78%|███████▊  | 7831/10000 [53:10<06:44,  5.36it/s]
 78%|███████▊  | 7832/10000 [53:11<06:43,  5.38it/s]
 78%|███████▊  | 7833/10000 [53:11<06:44,  5.36it/s]
 78%|███████▊  | 7834/10000 [53:11<06:47,  5.32it/s]
 78%|███████▊  | 7835/10000 [53:11<06:58,  5.17it/s]
 78%|███████▊  | 7836/10000 [53:11<07:00,  5.14it/s]
 78%|███████▊  | 7837/10000 [53:12<06:51,  5.25it/s]
 78%|███████▊  | 7838/10000 [53:12<06:53,  5.22it/s]
 78%|███████▊  | 7839/10000 [53:12<06:49,  5.28it/s]
 78%|███████▊  | 7840/10000 [53:12<06:43,  5.35it/s]
                                                    
{'loss': 0.076, 'grad_norm': 0.6337254047393799, 'learning_rate': 1.2233200303798158e-05}

 78%|███████▊  | 7840/10000 [53:12<06:43,  5.35it/s]
 78%|███████▊  | 7841/10000 [53:12<06:54,  5.21it/s]
 78%|███████▊  | 7842/10000 [53:12<06:56,  5.18it/s]
 78%|███████▊  | 7843/10000 [53:13<06:58,  5.15it/s]
 78%|███████▊  | 7844/10000 [53:13<07:01,  5.11it/s]
 78%|███████▊  | 7845/10000 [53:13<07:03,  5.09it/s]
 78%|███████▊  | 7846/10000 [53:13<07:00,  5.12it/s]
 78%|███████▊  | 7847/10000 [53:13<06:58,  5.15it/s]
 78%|███████▊  | 7848/10000 [53:14<06:51,  5.23it/s]
 78%|███████▊  | 7849/10000 [53:14<06:43,  5.33it/s]
 78%|███████▊  | 7850/10000 [53:14<06:40,  5.37it/s]
                                                    
{'loss': 0.0961, 'grad_norm': 0.5619378089904785, 'learning_rate': 1.2125048891951846e-05}

 78%|███████▊  | 7850/10000 [53:14<06:40,  5.37it/s]
 79%|███████▊  | 7851/10000 [53:14<06:41,  5.35it/s]
 79%|███████▊  | 7852/10000 [53:14<06:33,  5.46it/s]
 79%|███████▊  | 7853/10000 [53:15<06:35,  5.43it/s]
 79%|███████▊  | 7854/10000 [53:15<06:49,  5.24it/s]
 79%|███████▊  | 7855/10000 [53:15<06:55,  5.17it/s]
 79%|███████▊  | 7856/10000 [53:15<07:04,  5.05it/s]
 79%|███████▊  | 7857/10000 [53:15<07:10,  4.97it/s]
 79%|███████▊  | 7858/10000 [53:16<07:05,  5.04it/s]
 79%|███████▊  | 7859/10000 [53:16<06:54,  5.16it/s]
 79%|███████▊  | 7860/10000 [53:16<06:42,  5.31it/s]
                                                    
{'loss': 0.0629, 'grad_norm': 0.5772877931594849, 'learning_rate': 1.2017311674502745e-05}

 79%|███████▊  | 7860/10000 [53:16<06:42,  5.31it/s]
 79%|███████▊  | 7861/10000 [53:16<06:40,  5.34it/s]
 79%|███████▊  | 7862/10000 [53:16<06:34,  5.42it/s]
 79%|███████▊  | 7863/10000 [53:16<06:41,  5.32it/s]
 79%|███████▊  | 7864/10000 [53:17<06:53,  5.17it/s]
 79%|███████▊  | 7865/10000 [53:17<07:08,  4.99it/s]
 79%|███████▊  | 7866/10000 [53:17<07:26,  4.78it/s]
 79%|███████▊  | 7867/10000 [53:17<07:24,  4.80it/s]
 79%|███████▊  | 7868/10000 [53:18<07:15,  4.90it/s]
 79%|███████▊  | 7869/10000 [53:18<06:59,  5.08it/s]
 79%|███████▊  | 7870/10000 [53:18<06:50,  5.19it/s]
                                                    
{'loss': 0.0694, 'grad_norm': 0.48863834142684937, 'learning_rate': 1.1909989829647822e-05}

 79%|███████▊  | 7870/10000 [53:18<06:50,  5.19it/s]
 79%|███████▊  | 7871/10000 [53:18<06:53,  5.15it/s]
 79%|███████▊  | 7872/10000 [53:18<06:54,  5.13it/s]
 79%|███████▊  | 7873/10000 [53:19<07:02,  5.04it/s]
 79%|███████▊  | 7874/10000 [53:19<07:09,  4.95it/s]
 79%|███████▉  | 7875/10000 [53:19<07:08,  4.96it/s]
 79%|███████▉  | 7876/10000 [53:19<07:06,  4.98it/s]
 79%|███████▉  | 7877/10000 [53:19<07:01,  5.04it/s]
 79%|███████▉  | 7878/10000 [53:19<06:49,  5.18it/s]
 79%|███████▉  | 7879/10000 [53:20<06:46,  5.21it/s]
 79%|███████▉  | 7880/10000 [53:20<06:47,  5.20it/s]
                                                    
{'loss': 0.0642, 'grad_norm': 0.5381703972816467, 'learning_rate': 1.1803084531041553e-05}

 79%|███████▉  | 7880/10000 [53:20<06:47,  5.20it/s]
 79%|███████▉  | 7881/10000 [53:20<07:05,  4.98it/s]
 79%|███████▉  | 7882/10000 [53:20<07:20,  4.81it/s]
 79%|███████▉  | 7883/10000 [53:21<07:29,  4.71it/s]
 79%|███████▉  | 7884/10000 [53:21<07:21,  4.79it/s]
 79%|███████▉  | 7885/10000 [53:21<07:11,  4.90it/s]
 79%|███████▉  | 7886/10000 [53:21<06:53,  5.11it/s]
 79%|███████▉  | 7887/10000 [53:21<06:45,  5.21it/s]
 79%|███████▉  | 7888/10000 [53:21<06:40,  5.28it/s]
 79%|███████▉  | 7889/10000 [53:22<06:38,  5.29it/s]
 79%|███████▉  | 7890/10000 [53:22<06:45,  5.20it/s]
                                                    
{'loss': 0.0694, 'grad_norm': 0.667148232460022, 'learning_rate': 1.1696596947783162e-05}

 79%|███████▉  | 7890/10000 [53:22<06:45,  5.20it/s]
 79%|███████▉  | 7891/10000 [53:22<06:57,  5.06it/s]
 79%|███████▉  | 7892/10000 [53:22<07:01,  5.00it/s]
 79%|███████▉  | 7893/10000 [53:22<07:14,  4.85it/s]
 79%|███████▉  | 7894/10000 [53:23<07:07,  4.93it/s]
 79%|███████▉  | 7895/10000 [53:23<06:53,  5.09it/s]
 79%|███████▉  | 7896/10000 [53:23<06:52,  5.10it/s]
 79%|███████▉  | 7897/10000 [53:23<06:51,  5.11it/s]
 79%|███████▉  | 7898/10000 [53:23<06:48,  5.14it/s]
 79%|███████▉  | 7899/10000 [53:24<07:01,  4.99it/s]
 79%|███████▉  | 7900/10000 [53:24<07:03,  4.96it/s]
                                                    
{'loss': 0.0648, 'grad_norm': 0.6745719909667969, 'learning_rate': 1.1590528244403803e-05}

 79%|███████▉  | 7900/10000 [53:24<07:03,  4.96it/s]
 79%|███████▉  | 7901/10000 [53:24<07:12,  4.86it/s]
 79%|███████▉  | 7902/10000 [53:24<07:19,  4.77it/s]
 79%|███████▉  | 7903/10000 [53:25<07:33,  4.62it/s]
 79%|███████▉  | 7904/10000 [53:25<07:45,  4.51it/s]
 79%|███████▉  | 7905/10000 [53:25<08:18,  4.20it/s]
 79%|███████▉  | 7906/10000 [53:25<08:54,  3.92it/s]
 79%|███████▉  | 7907/10000 [53:26<09:07,  3.82it/s]
 79%|███████▉  | 7908/10000 [53:26<09:11,  3.79it/s]
 79%|███████▉  | 7909/10000 [53:26<08:56,  3.90it/s]
 79%|███████▉  | 7910/10000 [53:26<08:16,  4.21it/s]
                                                    
{'loss': 0.0724, 'grad_norm': 0.5412375926971436, 'learning_rate': 1.148487958085382e-05}

 79%|███████▉  | 7910/10000 [53:26<08:16,  4.21it/s]
 79%|███████▉  | 7911/10000 [53:27<08:06,  4.30it/s]
 79%|███████▉  | 7912/10000 [53:27<08:11,  4.25it/s]
 79%|███████▉  | 7913/10000 [53:27<08:06,  4.29it/s]
 79%|███████▉  | 7914/10000 [53:27<08:02,  4.33it/s]
 79%|███████▉  | 7915/10000 [53:27<08:13,  4.22it/s]
 79%|███████▉  | 7916/10000 [53:28<08:31,  4.08it/s]
 79%|███████▉  | 7917/10000 [53:28<08:06,  4.28it/s]
 79%|███████▉  | 7918/10000 [53:28<07:54,  4.39it/s]
 79%|███████▉  | 7919/10000 [53:28<07:37,  4.55it/s]
 79%|███████▉  | 7920/10000 [53:29<07:23,  4.69it/s]
                                                    
{'loss': 0.0777, 'grad_norm': 0.7747380137443542, 'learning_rate': 1.1379652112490086e-05}

 79%|███████▉  | 7920/10000 [53:29<07:23,  4.69it/s]
 79%|███████▉  | 7921/10000 [53:29<07:25,  4.67it/s]
 79%|███████▉  | 7922/10000 [53:29<07:37,  4.55it/s]
 79%|███████▉  | 7923/10000 [53:29<07:42,  4.49it/s]
 79%|███████▉  | 7924/10000 [53:29<07:49,  4.43it/s]
 79%|███████▉  | 7925/10000 [53:30<08:04,  4.28it/s]
 79%|███████▉  | 7926/10000 [53:30<08:08,  4.24it/s]
 79%|███████▉  | 7927/10000 [53:30<08:00,  4.32it/s]
 79%|███████▉  | 7928/10000 [53:30<07:59,  4.33it/s]
 79%|███████▉  | 7929/10000 [53:31<07:41,  4.49it/s]
 79%|███████▉  | 7930/10000 [53:31<07:28,  4.61it/s]
                                                    
{'loss': 0.0669, 'grad_norm': 0.6769585013389587, 'learning_rate': 1.1274846990063315e-05}

 79%|███████▉  | 7930/10000 [53:31<07:28,  4.61it/s]
 79%|███████▉  | 7931/10000 [53:31<07:35,  4.54it/s]
 79%|███████▉  | 7932/10000 [53:31<07:45,  4.44it/s]
 79%|███████▉  | 7933/10000 [53:32<08:02,  4.28it/s]
 79%|███████▉  | 7934/10000 [53:32<08:10,  4.21it/s]
 79%|███████▉  | 7935/10000 [53:32<08:26,  4.08it/s]
 79%|███████▉  | 7936/10000 [53:32<08:39,  3.97it/s]
 79%|███████▉  | 7937/10000 [53:33<08:16,  4.16it/s]
 79%|███████▉  | 7938/10000 [53:33<08:00,  4.29it/s]
 79%|███████▉  | 7939/10000 [53:33<07:42,  4.46it/s]
 79%|███████▉  | 7940/10000 [53:33<07:41,  4.47it/s]
                                                    
{'loss': 0.0721, 'grad_norm': 0.7341311573982239, 'learning_rate': 1.117046535970554e-05}

 79%|███████▉  | 7940/10000 [53:33<07:41,  4.47it/s]
 79%|███████▉  | 7941/10000 [53:33<07:58,  4.30it/s]
 79%|███████▉  | 7942/10000 [53:34<08:12,  4.18it/s]
 79%|███████▉  | 7943/10000 [53:34<08:21,  4.10it/s]
 79%|███████▉  | 7944/10000 [53:34<08:14,  4.16it/s]
 79%|███████▉  | 7945/10000 [53:34<08:01,  4.26it/s]
 79%|███████▉  | 7946/10000 [53:35<07:47,  4.40it/s]
 79%|███████▉  | 7947/10000 [53:35<07:37,  4.49it/s]
 79%|███████▉  | 7948/10000 [53:35<07:31,  4.54it/s]
 79%|███████▉  | 7949/10000 [53:35<07:33,  4.52it/s]
 80%|███████▉  | 7950/10000 [53:36<07:43,  4.42it/s]
                                                    
{'loss': 0.0595, 'grad_norm': 0.557282567024231, 'learning_rate': 1.106650836291755e-05}

 80%|███████▉  | 7950/10000 [53:36<07:43,  4.42it/s]
 80%|███████▉  | 7951/10000 [53:36<07:50,  4.36it/s]
 80%|███████▉  | 7952/10000 [53:36<07:40,  4.45it/s]
 80%|███████▉  | 7953/10000 [53:36<07:33,  4.51it/s]
 80%|███████▉  | 7954/10000 [53:36<07:33,  4.51it/s]
 80%|███████▉  | 7955/10000 [53:37<07:18,  4.67it/s]
 80%|███████▉  | 7956/10000 [53:37<07:14,  4.70it/s]
 80%|███████▉  | 7957/10000 [53:37<07:20,  4.64it/s]
 80%|███████▉  | 7958/10000 [53:37<07:32,  4.51it/s]
 80%|███████▉  | 7959/10000 [53:38<07:55,  4.29it/s]
 80%|███████▉  | 7960/10000 [53:38<08:26,  4.03it/s]
                                                    
{'loss': 0.0636, 'grad_norm': 0.6854016780853271, 'learning_rate': 1.0962977136556418e-05}

 80%|███████▉  | 7960/10000 [53:38<08:26,  4.03it/s]
 80%|███████▉  | 7961/10000 [53:38<08:25,  4.04it/s]
 80%|███████▉  | 7962/10000 [53:38<08:33,  3.97it/s]
 80%|███████▉  | 7963/10000 [53:39<07:58,  4.26it/s]
 80%|███████▉  | 7964/10000 [53:39<07:41,  4.41it/s]
 80%|███████▉  | 7965/10000 [53:39<07:37,  4.45it/s]
 80%|███████▉  | 7966/10000 [53:39<07:33,  4.49it/s]
 80%|███████▉  | 7967/10000 [53:39<07:42,  4.40it/s]
 80%|███████▉  | 7968/10000 [53:40<07:31,  4.50it/s]
 80%|███████▉  | 7969/10000 [53:40<07:26,  4.55it/s]
 80%|███████▉  | 7970/10000 [53:40<07:24,  4.57it/s]
                                                    
{'loss': 0.0785, 'grad_norm': 0.6598690748214722, 'learning_rate': 1.0859872812823024e-05}

 80%|███████▉  | 7970/10000 [53:40<07:24,  4.57it/s]
 80%|███████▉  | 7971/10000 [53:40<07:24,  4.57it/s]
 80%|███████▉  | 7972/10000 [53:40<07:16,  4.65it/s]
 80%|███████▉  | 7973/10000 [53:41<07:17,  4.63it/s]
 80%|███████▉  | 7974/10000 [53:41<07:17,  4.63it/s]
 80%|███████▉  | 7975/10000 [53:41<07:11,  4.69it/s]
 80%|███████▉  | 7976/10000 [53:41<07:10,  4.70it/s]
 80%|███████▉  | 7977/10000 [53:42<07:12,  4.68it/s]
 80%|███████▉  | 7978/10000 [53:42<07:10,  4.70it/s]
 80%|███████▉  | 7979/10000 [53:42<07:03,  4.78it/s]
 80%|███████▉  | 7980/10000 [53:42<06:58,  4.82it/s]
                                                    
{'loss': 0.0676, 'grad_norm': 0.687477707862854, 'learning_rate': 1.0757196519249747e-05}

 80%|███████▉  | 7980/10000 [53:42<06:58,  4.82it/s]
 80%|███████▉  | 7981/10000 [53:42<07:11,  4.68it/s]
 80%|███████▉  | 7982/10000 [53:43<07:25,  4.53it/s]
 80%|███████▉  | 7983/10000 [53:43<07:30,  4.48it/s]
 80%|███████▉  | 7984/10000 [53:43<07:55,  4.24it/s]
 80%|███████▉  | 7985/10000 [53:43<08:14,  4.07it/s]
 80%|███████▉  | 7986/10000 [53:44<08:29,  3.95it/s]
 80%|███████▉  | 7987/10000 [53:44<08:22,  4.01it/s]
 80%|███████▉  | 7988/10000 [53:44<08:23,  4.00it/s]
 80%|███████▉  | 7989/10000 [53:44<08:06,  4.13it/s]
 80%|███████▉  | 7990/10000 [53:45<07:53,  4.25it/s]
                                                    
{'loss': 0.0825, 'grad_norm': 0.7772767543792725, 'learning_rate': 1.0654949378688077e-05}

 80%|███████▉  | 7990/10000 [53:45<07:53,  4.25it/s]
 80%|███████▉  | 7991/10000 [53:45<07:42,  4.35it/s]
 80%|███████▉  | 7992/10000 [53:45<07:30,  4.46it/s]
 80%|███████▉  | 7993/10000 [53:45<07:36,  4.39it/s]
 80%|███████▉  | 7994/10000 [53:45<07:33,  4.42it/s]
 80%|███████▉  | 7995/10000 [53:46<07:37,  4.38it/s]
 80%|███████▉  | 7996/10000 [53:46<07:27,  4.48it/s]
 80%|███████▉  | 7997/10000 [53:46<07:24,  4.51it/s]
 80%|███████▉  | 7998/10000 [53:46<07:15,  4.60it/s]
 80%|███████▉  | 7999/10000 [53:47<07:04,  4.72it/s]
 80%|████████  | 8000/10000 [53:47<06:53,  4.84it/s]Rank 0, Worker 0: Caching shard...Rank 0, Worker 1: Caching shard...

Rank 0, Worker 4: Caching shard...Rank 0, Worker 2: Caching shard...

Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.73 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 19.18 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 19.19 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 19.31 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 19.34 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 19.45 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 8000: eval/loss = 0.0875 (37s)

                                                    
{'loss': 0.0875, 'grad_norm': 0.7754602432250977, 'learning_rate': 1.0553132509296376e-05}

 80%|████████  | 8000/10000 [54:24<06:53,  4.84it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-8000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-8000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-8000/wandb_config.json

 80%|████████  | 8001/10000 [54:49<10:30:25, 18.92s/it]
 80%|████████  | 8002/10000 [54:49<7:22:49, 13.30s/it] 
 80%|████████  | 8003/10000 [54:50<5:11:35,  9.36s/it]
 80%|████████  | 8004/10000 [54:50<3:39:45,  6.61s/it]
 80%|████████  | 8005/10000 [54:50<2:35:29,  4.68s/it]
 80%|████████  | 8006/10000 [54:50<1:50:29,  3.32s/it]
 80%|████████  | 8007/10000 [54:50<1:19:01,  2.38s/it]
 80%|████████  | 8008/10000 [54:51<57:01,  1.72s/it]  
 80%|████████  | 8009/10000 [54:51<41:39,  1.26s/it]
 80%|████████  | 8010/10000 [54:51<31:05,  1.07it/s]
                                                    
{'loss': 0.073, 'grad_norm': 0.6340537071228027, 'learning_rate': 1.0451747024527613e-05}

 80%|████████  | 8010/10000 [54:51<31:05,  1.07it/s]
 80%|████████  | 8011/10000 [54:51<23:51,  1.39it/s]
 80%|████████  | 8012/10000 [54:51<18:45,  1.77it/s]
 80%|████████  | 8013/10000 [54:52<15:21,  2.16it/s]
 80%|████████  | 8014/10000 [54:52<12:54,  2.56it/s]
 80%|████████  | 8015/10000 [54:52<11:01,  3.00it/s]
 80%|████████  | 8016/10000 [54:52<09:29,  3.48it/s]
 80%|████████  | 8017/10000 [54:52<08:23,  3.94it/s]
 80%|████████  | 8018/10000 [54:52<07:37,  4.34it/s]
 80%|████████  | 8019/10000 [54:53<07:07,  4.63it/s]
 80%|████████  | 8020/10000 [54:53<07:01,  4.70it/s]
                                                    
{'loss': 0.0723, 'grad_norm': 0.5363622307777405, 'learning_rate': 1.0350794033117189e-05}

 80%|████████  | 8020/10000 [54:53<07:01,  4.70it/s]
 80%|████████  | 8021/10000 [54:53<07:15,  4.54it/s]
 80%|████████  | 8022/10000 [54:53<07:09,  4.61it/s]
 80%|████████  | 8023/10000 [54:54<07:00,  4.70it/s]
 80%|████████  | 8024/10000 [54:54<06:54,  4.76it/s]
 80%|████████  | 8025/10000 [54:54<06:36,  4.99it/s]
 80%|████████  | 8026/10000 [54:54<06:20,  5.19it/s]
 80%|████████  | 8027/10000 [54:54<06:11,  5.30it/s]
 80%|████████  | 8028/10000 [54:54<06:05,  5.40it/s]
 80%|████████  | 8029/10000 [54:55<06:05,  5.39it/s]
 80%|████████  | 8030/10000 [54:55<06:25,  5.11it/s]
                                                    
{'loss': 0.059, 'grad_norm': 0.6165353059768677, 'learning_rate': 1.0250274639070856e-05}

 80%|████████  | 8030/10000 [54:55<06:25,  5.11it/s]
 80%|████████  | 8031/10000 [54:55<06:51,  4.78it/s]
 80%|████████  | 8032/10000 [54:55<06:59,  4.69it/s]
 80%|████████  | 8033/10000 [54:56<07:03,  4.64it/s]
 80%|████████  | 8034/10000 [54:56<06:43,  4.88it/s]
 80%|████████  | 8035/10000 [54:56<06:29,  5.05it/s]
 80%|████████  | 8036/10000 [54:56<06:17,  5.20it/s]
 80%|████████  | 8037/10000 [54:56<06:08,  5.33it/s]
 80%|████████  | 8038/10000 [54:56<06:18,  5.18it/s]
 80%|████████  | 8039/10000 [54:57<06:27,  5.06it/s]
 80%|████████  | 8040/10000 [54:57<06:35,  4.96it/s]
                                                    
{'loss': 0.0642, 'grad_norm': 0.6290779113769531, 'learning_rate': 1.0150189941652599e-05}

 80%|████████  | 8040/10000 [54:57<06:35,  4.96it/s]
 80%|████████  | 8041/10000 [54:57<06:54,  4.72it/s]
 80%|████████  | 8042/10000 [54:57<06:57,  4.69it/s]
 80%|████████  | 8043/10000 [54:58<06:42,  4.87it/s]
 80%|████████  | 8044/10000 [54:58<06:27,  5.05it/s]
 80%|████████  | 8045/10000 [54:58<06:13,  5.23it/s]
 80%|████████  | 8046/10000 [54:58<06:11,  5.26it/s]
 80%|████████  | 8047/10000 [54:58<06:23,  5.10it/s]
 80%|████████  | 8048/10000 [54:58<06:37,  4.91it/s]
 80%|████████  | 8049/10000 [54:59<06:43,  4.84it/s]
 80%|████████  | 8050/10000 [54:59<07:08,  4.55it/s]
                                                    
{'loss': 0.0716, 'grad_norm': 0.6127679944038391, 'learning_rate': 1.0050541035372635e-05}

 80%|████████  | 8050/10000 [54:59<07:08,  4.55it/s]
 81%|████████  | 8051/10000 [54:59<06:53,  4.71it/s]
 81%|████████  | 8052/10000 [54:59<06:34,  4.94it/s]
 81%|████████  | 8053/10000 [55:00<06:20,  5.12it/s]
 81%|████████  | 8054/10000 [55:00<06:28,  5.02it/s]
 81%|████████  | 8055/10000 [55:00<06:32,  4.95it/s]
 81%|████████  | 8056/10000 [55:00<06:45,  4.79it/s]
 81%|████████  | 8057/10000 [55:00<06:48,  4.75it/s]
 81%|████████  | 8058/10000 [55:01<06:41,  4.84it/s]
 81%|████████  | 8059/10000 [55:01<06:29,  4.99it/s]
 81%|████████  | 8060/10000 [55:01<06:14,  5.18it/s]
                                                    
{'loss': 0.0842, 'grad_norm': 0.669055700302124, 'learning_rate': 9.951329009975458e-06}

 81%|████████  | 8060/10000 [55:01<06:14,  5.18it/s]
 81%|████████  | 8061/10000 [55:01<06:13,  5.19it/s]
 81%|████████  | 8062/10000 [55:01<06:13,  5.18it/s]
 81%|████████  | 8063/10000 [55:02<07:54,  4.08it/s]
 81%|████████  | 8064/10000 [55:02<07:30,  4.30it/s]
 81%|████████  | 8065/10000 [55:02<07:14,  4.45it/s]
 81%|████████  | 8066/10000 [55:02<06:58,  4.62it/s]
 81%|████████  | 8067/10000 [55:03<06:59,  4.60it/s]
 81%|████████  | 8068/10000 [55:03<06:54,  4.66it/s]
 81%|████████  | 8069/10000 [55:03<07:01,  4.58it/s]
 81%|████████  | 8070/10000 [55:03<06:53,  4.66it/s]
                                                    
{'loss': 0.08, 'grad_norm': 0.7711713314056396, 'learning_rate': 9.852554950427845e-06}

 81%|████████  | 8070/10000 [55:03<06:53,  4.66it/s]
 81%|████████  | 8071/10000 [55:03<07:00,  4.59it/s]
 81%|████████  | 8072/10000 [55:04<07:00,  4.59it/s]
 81%|████████  | 8073/10000 [55:04<06:52,  4.67it/s]
 81%|████████  | 8074/10000 [55:04<06:43,  4.77it/s]
 81%|████████  | 8075/10000 [55:04<06:28,  4.95it/s]
 81%|████████  | 8076/10000 [55:04<06:16,  5.11it/s]
 81%|████████  | 8077/10000 [55:05<06:15,  5.13it/s]
 81%|████████  | 8078/10000 [55:05<06:21,  5.04it/s]
 81%|████████  | 8079/10000 [55:05<06:32,  4.89it/s]
 81%|████████  | 8080/10000 [55:05<06:35,  4.86it/s]
                                                    
{'loss': 0.0588, 'grad_norm': 0.6247637271881104, 'learning_rate': 9.754219936907105e-06}

 81%|████████  | 8080/10000 [55:05<06:35,  4.86it/s]
 81%|████████  | 8081/10000 [55:05<06:47,  4.71it/s]
 81%|████████  | 8082/10000 [55:06<06:44,  4.74it/s]
 81%|████████  | 8083/10000 [55:06<06:38,  4.81it/s]
 81%|████████  | 8084/10000 [55:06<06:29,  4.91it/s]
 81%|████████  | 8085/10000 [55:06<06:22,  5.00it/s]
 81%|████████  | 8086/10000 [55:06<06:16,  5.08it/s]
 81%|████████  | 8087/10000 [55:07<06:20,  5.03it/s]
 81%|████████  | 8088/10000 [55:07<06:23,  4.98it/s]
 81%|████████  | 8089/10000 [55:07<06:40,  4.77it/s]
 81%|████████  | 8090/10000 [55:07<06:44,  4.72it/s]
                                                    
{'loss': 0.0629, 'grad_norm': 0.5155637860298157, 'learning_rate': 9.656325044789194e-06}

 81%|████████  | 8090/10000 [55:07<06:44,  4.72it/s]
 81%|████████  | 8091/10000 [55:08<07:04,  4.49it/s]
 81%|████████  | 8092/10000 [55:08<07:21,  4.32it/s]
 81%|████████  | 8093/10000 [55:08<07:15,  4.38it/s]
 81%|████████  | 8094/10000 [55:08<07:16,  4.37it/s]
 81%|████████  | 8095/10000 [55:08<07:03,  4.49it/s]
 81%|████████  | 8096/10000 [55:09<07:02,  4.50it/s]
 81%|████████  | 8097/10000 [55:09<07:10,  4.42it/s]
 81%|████████  | 8098/10000 [55:09<07:32,  4.21it/s]
 81%|████████  | 8099/10000 [55:09<07:41,  4.12it/s]
 81%|████████  | 8100/10000 [55:10<07:57,  3.98it/s]
                                                    
{'loss': 0.0629, 'grad_norm': 0.6250995397567749, 'learning_rate': 9.55887134463697e-06}

 81%|████████  | 8100/10000 [55:10<07:57,  3.98it/s]
 81%|████████  | 8101/10000 [55:10<07:57,  3.98it/s]
 81%|████████  | 8102/10000 [55:10<07:47,  4.06it/s]
 81%|████████  | 8103/10000 [55:10<07:43,  4.09it/s]
 81%|████████  | 8104/10000 [55:11<07:32,  4.19it/s]
 81%|████████  | 8105/10000 [55:11<07:21,  4.29it/s]
 81%|████████  | 8106/10000 [55:11<07:21,  4.29it/s]
 81%|████████  | 8107/10000 [55:11<07:19,  4.31it/s]
 81%|████████  | 8108/10000 [55:12<07:29,  4.20it/s]
 81%|████████  | 8109/10000 [55:12<07:37,  4.14it/s]
 81%|████████  | 8110/10000 [55:12<07:45,  4.06it/s]
                                                    
{'loss': 0.0658, 'grad_norm': 0.6513325572013855, 'learning_rate': 9.461859902188475e-06}

 81%|████████  | 8110/10000 [55:12<07:45,  4.06it/s]
 81%|████████  | 8111/10000 [55:12<07:43,  4.07it/s]
 81%|████████  | 8112/10000 [55:13<07:35,  4.15it/s]
 81%|████████  | 8113/10000 [55:13<07:42,  4.08it/s]
 81%|████████  | 8114/10000 [55:13<07:33,  4.16it/s]
 81%|████████  | 8115/10000 [55:13<07:20,  4.28it/s]
 81%|████████  | 8116/10000 [55:13<07:02,  4.46it/s]
 81%|████████  | 8117/10000 [55:14<06:50,  4.59it/s]
 81%|████████  | 8118/10000 [55:14<07:01,  4.47it/s]
 81%|████████  | 8119/10000 [55:14<07:09,  4.38it/s]
 81%|████████  | 8120/10000 [55:14<07:17,  4.30it/s]
                                                    
{'loss': 0.091, 'grad_norm': 0.5469868779182434, 'learning_rate': 9.365291778345303e-06}

 81%|████████  | 8120/10000 [55:14<07:17,  4.30it/s]
 81%|████████  | 8121/10000 [55:15<07:30,  4.17it/s]
 81%|████████  | 8122/10000 [55:15<07:26,  4.20it/s]
 81%|████████  | 8123/10000 [55:15<07:12,  4.34it/s]
 81%|████████  | 8124/10000 [55:15<07:02,  4.44it/s]
 81%|████████▏ | 8125/10000 [55:15<06:53,  4.54it/s]
 81%|████████▏ | 8126/10000 [55:16<06:56,  4.50it/s]
 81%|████████▏ | 8127/10000 [55:16<07:04,  4.41it/s]
 81%|████████▏ | 8128/10000 [55:16<06:55,  4.50it/s]
 81%|████████▏ | 8129/10000 [55:16<06:57,  4.48it/s]
 81%|████████▏ | 8130/10000 [55:17<06:53,  4.53it/s]
                                                    
{'loss': 0.0812, 'grad_norm': 0.7504246830940247, 'learning_rate': 9.269168029160991e-06}

 81%|████████▏ | 8130/10000 [55:17<06:53,  4.53it/s]
 81%|████████▏ | 8131/10000 [55:17<06:52,  4.53it/s]
 81%|████████▏ | 8132/10000 [55:17<06:42,  4.64it/s]
 81%|████████▏ | 8133/10000 [55:17<06:38,  4.69it/s]
 81%|████████▏ | 8134/10000 [55:17<06:34,  4.73it/s]
 81%|████████▏ | 8135/10000 [55:18<06:34,  4.72it/s]
 81%|████████▏ | 8136/10000 [55:18<06:32,  4.75it/s]
 81%|████████▏ | 8137/10000 [55:18<06:42,  4.63it/s]
 81%|████████▏ | 8138/10000 [55:18<07:00,  4.43it/s]
 81%|████████▏ | 8139/10000 [55:19<06:45,  4.59it/s]
 81%|████████▏ | 8140/10000 [55:19<06:40,  4.64it/s]
                                                    
{'loss': 0.0706, 'grad_norm': 0.6921966671943665, 'learning_rate': 9.173489705829447e-06}

 81%|████████▏ | 8140/10000 [55:19<06:40,  4.64it/s]
 81%|████████▏ | 8141/10000 [55:19<06:31,  4.75it/s]
 81%|████████▏ | 8142/10000 [55:19<06:24,  4.84it/s]
 81%|████████▏ | 8143/10000 [55:19<06:23,  4.84it/s]
 81%|████████▏ | 8144/10000 [55:20<06:22,  4.86it/s]
 81%|████████▏ | 8145/10000 [55:20<06:25,  4.81it/s]
 81%|████████▏ | 8146/10000 [55:20<06:40,  4.63it/s]
 81%|████████▏ | 8147/10000 [55:20<06:55,  4.46it/s]
 81%|████████▏ | 8148/10000 [55:20<06:51,  4.50it/s]
 81%|████████▏ | 8149/10000 [55:21<06:46,  4.56it/s]
 82%|████████▏ | 8150/10000 [55:21<06:31,  4.73it/s]
                                                    
{'loss': 0.0749, 'grad_norm': 0.6971414089202881, 'learning_rate': 9.078257854673516e-06}

 82%|████████▏ | 8150/10000 [55:21<06:31,  4.73it/s]
 82%|████████▏ | 8151/10000 [55:21<06:25,  4.79it/s]
 82%|████████▏ | 8152/10000 [55:21<06:20,  4.85it/s]
 82%|████████▏ | 8153/10000 [55:21<06:20,  4.85it/s]
 82%|████████▏ | 8154/10000 [55:22<06:27,  4.76it/s]
 82%|████████▏ | 8155/10000 [55:22<06:27,  4.76it/s]
 82%|████████▏ | 8156/10000 [55:22<06:34,  4.68it/s]
 82%|████████▏ | 8157/10000 [55:22<06:21,  4.84it/s]
 82%|████████▏ | 8158/10000 [55:22<06:09,  4.98it/s]
 82%|████████▏ | 8159/10000 [55:23<05:57,  5.15it/s]
 82%|████████▏ | 8160/10000 [55:23<06:01,  5.09it/s]
                                                    
{'loss': 0.0684, 'grad_norm': 0.5341566205024719, 'learning_rate': 8.983473517133429e-06}

 82%|████████▏ | 8160/10000 [55:23<06:01,  5.09it/s]
 82%|████████▏ | 8161/10000 [55:23<06:12,  4.94it/s]
 82%|████████▏ | 8162/10000 [55:23<06:13,  4.92it/s]
 82%|████████▏ | 8163/10000 [55:24<06:11,  4.94it/s]
 82%|████████▏ | 8164/10000 [55:24<06:08,  4.98it/s]
 82%|████████▏ | 8165/10000 [55:24<06:00,  5.10it/s]
 82%|████████▏ | 8166/10000 [55:24<05:54,  5.18it/s]
 82%|████████▏ | 8167/10000 [55:24<05:47,  5.28it/s]
 82%|████████▏ | 8168/10000 [55:24<05:44,  5.31it/s]
 82%|████████▏ | 8169/10000 [55:25<05:43,  5.32it/s]
 82%|████████▏ | 8170/10000 [55:25<05:41,  5.36it/s]
                                                    
{'loss': 0.0779, 'grad_norm': 0.8540531992912292, 'learning_rate': 8.889137729755537e-06}

 82%|████████▏ | 8170/10000 [55:25<05:41,  5.36it/s]
 82%|████████▏ | 8171/10000 [55:25<05:45,  5.30it/s]
 82%|████████▏ | 8172/10000 [55:25<05:43,  5.32it/s]
 82%|████████▏ | 8173/10000 [55:25<05:41,  5.36it/s]
 82%|████████▏ | 8174/10000 [55:26<05:39,  5.38it/s]
 82%|████████▏ | 8175/10000 [55:26<05:40,  5.36it/s]
 82%|████████▏ | 8176/10000 [55:26<05:41,  5.34it/s]
 82%|████████▏ | 8177/10000 [55:26<05:34,  5.45it/s]
 82%|████████▏ | 8178/10000 [55:26<05:31,  5.49it/s]
 82%|████████▏ | 8179/10000 [55:26<05:27,  5.56it/s]
 82%|████████▏ | 8180/10000 [55:27<05:24,  5.60it/s]
                                                    
{'loss': 0.0682, 'grad_norm': 0.6469535827636719, 'learning_rate': 8.79525152418087e-06}

 82%|████████▏ | 8180/10000 [55:27<05:24,  5.60it/s]
 82%|████████▏ | 8181/10000 [55:27<05:36,  5.40it/s]
 82%|████████▏ | 8182/10000 [55:27<05:36,  5.40it/s]
 82%|████████▏ | 8183/10000 [55:27<05:36,  5.41it/s]
 82%|████████▏ | 8184/10000 [55:27<05:33,  5.45it/s]
 82%|████████▏ | 8185/10000 [55:28<05:28,  5.52it/s]
 82%|████████▏ | 8186/10000 [55:28<05:25,  5.58it/s]
 82%|████████▏ | 8187/10000 [55:28<05:30,  5.49it/s]
 82%|████████▏ | 8188/10000 [55:28<05:30,  5.48it/s]
 82%|████████▏ | 8189/10000 [55:28<05:30,  5.47it/s]
 82%|████████▏ | 8190/10000 [55:28<05:34,  5.42it/s]
                                                    
{'loss': 0.0793, 'grad_norm': 0.6491568684577942, 'learning_rate': 8.701815927133961e-06}

 82%|████████▏ | 8190/10000 [55:29<05:34,  5.42it/s]
 82%|████████▏ | 8191/10000 [55:29<05:40,  5.31it/s]
 82%|████████▏ | 8192/10000 [55:29<05:35,  5.39it/s]
 82%|████████▏ | 8193/10000 [55:29<05:31,  5.44it/s]
 82%|████████▏ | 8194/10000 [55:29<05:28,  5.50it/s]
 82%|████████▏ | 8195/10000 [55:29<05:29,  5.48it/s]
 82%|████████▏ | 8196/10000 [55:30<05:38,  5.32it/s]
 82%|████████▏ | 8197/10000 [55:30<06:01,  4.99it/s]
 82%|████████▏ | 8198/10000 [55:30<06:13,  4.83it/s]
 82%|████████▏ | 8199/10000 [55:30<06:08,  4.89it/s]
 82%|████████▏ | 8200/10000 [55:30<06:06,  4.91it/s]
                                                    
{'loss': 0.0693, 'grad_norm': 0.6318727135658264, 'learning_rate': 8.608831960411534e-06}

 82%|████████▏ | 8200/10000 [55:30<06:06,  4.91it/s]
 82%|████████▏ | 8201/10000 [55:31<06:08,  4.88it/s]
 82%|████████▏ | 8202/10000 [55:31<05:54,  5.07it/s]
 82%|████████▏ | 8203/10000 [55:31<05:43,  5.23it/s]
 82%|████████▏ | 8204/10000 [55:31<05:37,  5.33it/s]
 82%|████████▏ | 8205/10000 [55:31<05:36,  5.33it/s]
 82%|████████▏ | 8206/10000 [55:32<05:40,  5.27it/s]
 82%|████████▏ | 8207/10000 [55:32<05:46,  5.18it/s]
 82%|████████▏ | 8208/10000 [55:32<05:48,  5.14it/s]
 82%|████████▏ | 8209/10000 [55:32<05:47,  5.15it/s]
 82%|████████▏ | 8210/10000 [55:32<05:43,  5.21it/s]
                                                    
{'loss': 0.0798, 'grad_norm': 0.6577479243278503, 'learning_rate': 8.516300640871321e-06}

 82%|████████▏ | 8210/10000 [55:32<05:43,  5.21it/s]
 82%|████████▏ | 8211/10000 [55:33<05:41,  5.24it/s]
 82%|████████▏ | 8212/10000 [55:33<05:32,  5.38it/s]
 82%|████████▏ | 8213/10000 [55:33<05:38,  5.27it/s]
 82%|████████▏ | 8214/10000 [55:33<06:02,  4.92it/s]
 82%|████████▏ | 8215/10000 [55:33<05:54,  5.03it/s]
 82%|████████▏ | 8216/10000 [55:34<05:52,  5.06it/s]
 82%|████████▏ | 8217/10000 [55:34<05:48,  5.12it/s]
 82%|████████▏ | 8218/10000 [55:34<05:44,  5.17it/s]
 82%|████████▏ | 8219/10000 [55:34<05:43,  5.19it/s]
 82%|████████▏ | 8220/10000 [55:34<05:43,  5.18it/s]
                                                    
{'loss': 0.0761, 'grad_norm': 0.7536382079124451, 'learning_rate': 8.424222980421038e-06}

 82%|████████▏ | 8220/10000 [55:34<05:43,  5.18it/s]
 82%|████████▏ | 8221/10000 [55:34<05:44,  5.16it/s]
 82%|████████▏ | 8222/10000 [55:35<05:42,  5.19it/s]
 82%|████████▏ | 8223/10000 [55:35<05:41,  5.20it/s]
 82%|████████▏ | 8224/10000 [55:35<05:39,  5.24it/s]
 82%|████████▏ | 8225/10000 [55:35<05:42,  5.19it/s]
 82%|████████▏ | 8226/10000 [55:35<05:43,  5.16it/s]
 82%|████████▏ | 8227/10000 [55:36<05:44,  5.14it/s]
 82%|████████▏ | 8228/10000 [55:36<05:53,  5.01it/s]
 82%|████████▏ | 8229/10000 [55:36<05:52,  5.02it/s]
 82%|████████▏ | 8230/10000 [55:36<05:56,  4.97it/s]
                                                    
{'loss': 0.0709, 'grad_norm': 0.6986160278320312, 'learning_rate': 8.332599986007184e-06}

 82%|████████▏ | 8230/10000 [55:36<05:56,  4.97it/s]
 82%|████████▏ | 8231/10000 [55:36<05:57,  4.95it/s]
 82%|████████▏ | 8232/10000 [55:37<05:44,  5.14it/s]
 82%|████████▏ | 8233/10000 [55:37<05:41,  5.18it/s]
 82%|████████▏ | 8234/10000 [55:37<05:39,  5.20it/s]
 82%|████████▏ | 8235/10000 [55:37<05:35,  5.26it/s]
 82%|████████▏ | 8236/10000 [55:37<05:36,  5.23it/s]
 82%|████████▏ | 8237/10000 [55:38<05:45,  5.10it/s]
 82%|████████▏ | 8238/10000 [55:38<05:50,  5.02it/s]
 82%|████████▏ | 8239/10000 [55:38<05:50,  5.02it/s]
 82%|████████▏ | 8240/10000 [55:38<05:49,  5.03it/s]
                                                    
{'loss': 0.0887, 'grad_norm': 0.5849621891975403, 'learning_rate': 8.241432659604203e-06}

 82%|████████▏ | 8240/10000 [55:38<05:49,  5.03it/s]
 82%|████████▏ | 8241/10000 [55:38<05:51,  5.00it/s]
 82%|████████▏ | 8242/10000 [55:39<05:49,  5.04it/s]
 82%|████████▏ | 8243/10000 [55:39<05:44,  5.10it/s]
 82%|████████▏ | 8244/10000 [55:39<05:39,  5.18it/s]
 82%|████████▏ | 8245/10000 [55:39<05:47,  5.05it/s]
 82%|████████▏ | 8246/10000 [55:39<05:46,  5.06it/s]
 82%|████████▏ | 8247/10000 [55:40<05:47,  5.04it/s]
 82%|████████▏ | 8248/10000 [55:40<05:54,  4.94it/s]
 82%|████████▏ | 8249/10000 [55:40<06:05,  4.80it/s]
 82%|████████▎ | 8250/10000 [55:40<06:05,  4.79it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 1: Caching shard...Rank 0, Worker 4: Caching shard...

Rank 0, Worker 5: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.69 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.23 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.27 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.27 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.29 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.37 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 8250: eval/loss = 0.0894 (35s)

                                                    
{'loss': 0.075, 'grad_norm': 0.7751139402389526, 'learning_rate': 8.150721998203331e-06}

 82%|████████▎ | 8250/10000 [56:15<06:05,  4.79it/s]
 83%|████████▎ | 8251/10000 [56:16<5:13:37, 10.76s/it]
 83%|████████▎ | 8252/10000 [56:16<3:40:57,  7.58s/it]
 83%|████████▎ | 8253/10000 [56:16<2:36:08,  5.36s/it]
 83%|████████▎ | 8254/10000 [56:16<1:50:45,  3.81s/it]
 83%|████████▎ | 8255/10000 [56:16<1:19:27,  2.73s/it]
 83%|████████▎ | 8256/10000 [56:17<57:43,  1.99s/it]  
 83%|████████▎ | 8257/10000 [56:17<42:06,  1.45s/it]
 83%|████████▎ | 8258/10000 [56:17<31:18,  1.08s/it]
 83%|████████▎ | 8259/10000 [56:17<23:33,  1.23it/s]
 83%|████████▎ | 8260/10000 [56:17<18:00,  1.61it/s]
                                                    
{'loss': 0.0777, 'grad_norm': 0.6930674314498901, 'learning_rate': 8.06046899380184e-06}

 83%|████████▎ | 8260/10000 [56:17<18:00,  1.61it/s]
 83%|████████▎ | 8261/10000 [56:18<14:12,  2.04it/s]
 83%|████████▎ | 8262/10000 [56:18<11:35,  2.50it/s]
 83%|████████▎ | 8263/10000 [56:18<09:44,  2.97it/s]
 83%|████████▎ | 8264/10000 [56:18<08:31,  3.39it/s]
 83%|████████▎ | 8265/10000 [56:18<07:41,  3.76it/s]
 83%|████████▎ | 8266/10000 [56:19<07:02,  4.10it/s]
 83%|████████▎ | 8267/10000 [56:19<06:38,  4.35it/s]
 83%|████████▎ | 8268/10000 [56:19<06:12,  4.65it/s]
 83%|████████▎ | 8269/10000 [56:19<05:51,  4.92it/s]
 83%|████████▎ | 8270/10000 [56:19<05:38,  5.11it/s]
                                                    
{'loss': 0.0775, 'grad_norm': 0.5731142163276672, 'learning_rate': 7.970674633392133e-06}

 83%|████████▎ | 8270/10000 [56:19<05:38,  5.11it/s]Rank 0, Worker 2: Wait for shard 36 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 83%|████████▎ | 8271/10000 [56:19<05:41,  5.06it/s]
 83%|████████▎ | 8272/10000 [56:20<05:37,  5.12it/s]
 83%|████████▎ | 8273/10000 [56:20<05:37,  5.11it/s]
 83%|████████▎ | 8274/10000 [56:20<05:42,  5.04it/s]
 83%|████████▎ | 8275/10000 [56:20<05:43,  5.02it/s]
 83%|████████▎ | 8276/10000 [56:20<05:37,  5.11it/s]
 83%|████████▎ | 8277/10000 [56:21<05:27,  5.25it/s]
 83%|████████▎ | 8278/10000 [56:21<05:20,  5.37it/s]
 83%|████████▎ | 8279/10000 [56:21<05:16,  5.44it/s]
 83%|████████▎ | 8280/10000 [56:21<05:11,  5.52it/s]
                                                    
{'loss': 0.0572, 'grad_norm': 0.6388645172119141, 'learning_rate': 7.881339898950924e-06}

 83%|████████▎ | 8280/10000 [56:21<05:11,  5.52it/s]
 83%|████████▎ | 8281/10000 [56:21<05:16,  5.44it/s]Rank 0, Worker 1: Wait for shard 7 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 83%|████████▎ | 8282/10000 [56:22<05:24,  5.29it/s]
 83%|████████▎ | 8283/10000 [56:22<05:29,  5.22it/s]
 83%|████████▎ | 8284/10000 [56:22<05:31,  5.17it/s]
 83%|████████▎ | 8285/10000 [56:22<05:32,  5.16it/s]
 83%|████████▎ | 8286/10000 [56:22<05:30,  5.19it/s]
 83%|████████▎ | 8287/10000 [56:23<05:22,  5.31it/s]
 83%|████████▎ | 8288/10000 [56:23<05:20,  5.34it/s]
 83%|████████▎ | 8289/10000 [56:23<05:18,  5.37it/s]
 83%|████████▎ | 8290/10000 [56:23<05:28,  5.21it/s]
                                                    
{'loss': 0.0599, 'grad_norm': 0.7337496876716614, 'learning_rate': 7.792465767428597e-06}

 83%|████████▎ | 8290/10000 [56:23<05:28,  5.21it/s]
 83%|████████▎ | 8291/10000 [56:23<05:40,  5.02it/s]
 83%|████████▎ | 8292/10000 [56:24<05:41,  5.00it/s]
 83%|████████▎ | 8293/10000 [56:24<05:40,  5.01it/s]
 83%|████████▎ | 8294/10000 [56:24<05:34,  5.11it/s]
 83%|████████▎ | 8295/10000 [56:24<05:26,  5.22it/s]
 83%|████████▎ | 8296/10000 [56:24<05:21,  5.30it/s]
 83%|████████▎ | 8297/10000 [56:24<05:22,  5.28it/s]
 83%|████████▎ | 8298/10000 [56:25<05:23,  5.27it/s]
 83%|████████▎ | 8299/10000 [56:25<05:28,  5.19it/s]
 83%|████████▎ | 8300/10000 [56:25<05:35,  5.06it/s]
                                                    
{'loss': 0.0653, 'grad_norm': 0.5776150226593018, 'learning_rate': 7.704053210738376e-06}

 83%|████████▎ | 8300/10000 [56:25<05:35,  5.06it/s]
 83%|████████▎ | 8301/10000 [56:25<05:38,  5.02it/s]
 83%|████████▎ | 8302/10000 [56:25<05:44,  4.93it/s]Rank 0, Worker 4: Wait for shard 52 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 83%|████████▎ | 8303/10000 [56:26<05:58,  4.74it/s]
 83%|████████▎ | 8304/10000 [56:26<05:52,  4.82it/s]
 83%|████████▎ | 8305/10000 [56:26<06:00,  4.70it/s]
 83%|████████▎ | 8306/10000 [56:26<05:56,  4.75it/s]
 83%|████████▎ | 8307/10000 [56:27<05:53,  4.79it/s]
 83%|████████▎ | 8308/10000 [56:27<05:56,  4.75it/s]
 83%|████████▎ | 8309/10000 [56:27<05:58,  4.72it/s]
 83%|████████▎ | 8310/10000 [56:27<06:02,  4.66it/s]
                                                    
{'loss': 0.0562, 'grad_norm': 0.6158093214035034, 'learning_rate': 7.6161031957458494e-06}

 83%|████████▎ | 8310/10000 [56:27<06:02,  4.66it/s]
 83%|████████▎ | 8311/10000 [56:27<06:12,  4.54it/s]
 83%|████████▎ | 8312/10000 [56:28<06:11,  4.54it/s]
 83%|████████▎ | 8313/10000 [56:28<06:08,  4.57it/s]
 83%|████████▎ | 8314/10000 [56:28<05:51,  4.80it/s]
 83%|████████▎ | 8315/10000 [56:28<05:40,  4.94it/s]
 83%|████████▎ | 8316/10000 [56:28<05:38,  4.97it/s]
 83%|████████▎ | 8317/10000 [56:29<05:35,  5.02it/s]
 83%|████████▎ | 8318/10000 [56:29<05:42,  4.91it/s]
 83%|████████▎ | 8319/10000 [56:29<05:52,  4.77it/s]
 83%|████████▎ | 8320/10000 [56:29<05:56,  4.72it/s]
                                                    
{'loss': 0.0688, 'grad_norm': 0.6314040422439575, 'learning_rate': 7.5286166842582605e-06}

 83%|████████▎ | 8320/10000 [56:29<05:56,  4.72it/s]
 83%|████████▎ | 8321/10000 [56:29<06:01,  4.64it/s]
 83%|████████▎ | 8322/10000 [56:30<05:46,  4.84it/s]
 83%|████████▎ | 8323/10000 [56:30<05:33,  5.03it/s]
 83%|████████▎ | 8324/10000 [56:30<05:26,  5.13it/s]
 83%|████████▎ | 8325/10000 [56:30<05:19,  5.25it/s]
 83%|████████▎ | 8326/10000 [56:30<05:19,  5.24it/s]
 83%|████████▎ | 8327/10000 [56:31<05:37,  4.96it/s]
 83%|████████▎ | 8328/10000 [56:31<05:42,  4.89it/s]
 83%|████████▎ | 8329/10000 [56:31<05:49,  4.79it/s]
 83%|████████▎ | 8330/10000 [56:31<05:57,  4.67it/s]
                                                    
{'loss': 0.0766, 'grad_norm': 0.6137526035308838, 'learning_rate': 7.4415946330140814e-06}

 83%|████████▎ | 8330/10000 [56:31<05:57,  4.67it/s]
 83%|████████▎ | 8331/10000 [56:32<06:02,  4.61it/s]
 83%|████████▎ | 8332/10000 [56:32<05:49,  4.78it/s]
 83%|████████▎ | 8333/10000 [56:32<05:40,  4.90it/s]
 83%|████████▎ | 8334/10000 [56:32<05:34,  4.98it/s]
 83%|████████▎ | 8335/10000 [56:32<05:33,  4.99it/s]
 83%|████████▎ | 8336/10000 [56:33<05:45,  4.82it/s]
 83%|████████▎ | 8337/10000 [56:33<05:45,  4.82it/s]
 83%|████████▎ | 8338/10000 [56:33<05:47,  4.79it/s]
 83%|████████▎ | 8339/10000 [56:33<05:43,  4.84it/s]
 83%|████████▎ | 8340/10000 [56:33<05:41,  4.86it/s]
                                                    
{'loss': 0.0651, 'grad_norm': 0.5677177309989929, 'learning_rate': 7.3550379936725644e-06}

 83%|████████▎ | 8340/10000 [56:33<05:41,  4.86it/s]
 83%|████████▎ | 8341/10000 [56:34<05:41,  4.86it/s]
 83%|████████▎ | 8342/10000 [56:34<05:31,  5.00it/s]
 83%|████████▎ | 8343/10000 [56:34<05:31,  4.99it/s]
 83%|████████▎ | 8344/10000 [56:34<05:27,  5.05it/s]
 83%|████████▎ | 8345/10000 [56:34<05:24,  5.10it/s]
 83%|████████▎ | 8346/10000 [56:35<05:23,  5.11it/s]
 83%|████████▎ | 8347/10000 [56:35<05:20,  5.15it/s]
 83%|████████▎ | 8348/10000 [56:35<05:23,  5.10it/s]
 83%|████████▎ | 8349/10000 [56:35<05:28,  5.03it/s]
 84%|████████▎ | 8350/10000 [56:35<05:49,  4.72it/s]
                                                    
{'loss': 0.0872, 'grad_norm': 0.6850761771202087, 'learning_rate': 7.2689477128032035e-06}

 84%|████████▎ | 8350/10000 [56:35<05:49,  4.72it/s]
 84%|████████▎ | 8351/10000 [56:36<05:57,  4.62it/s]
 84%|████████▎ | 8352/10000 [56:36<06:02,  4.54it/s]
 84%|████████▎ | 8353/10000 [56:36<05:50,  4.69it/s]
 84%|████████▎ | 8354/10000 [56:36<05:39,  4.85it/s]
 84%|████████▎ | 8355/10000 [56:36<05:54,  4.64it/s]
 84%|████████▎ | 8356/10000 [56:37<05:53,  4.66it/s]
 84%|████████▎ | 8357/10000 [56:37<05:43,  4.78it/s]
 84%|████████▎ | 8358/10000 [56:37<05:41,  4.81it/s]
 84%|████████▎ | 8359/10000 [56:37<05:37,  4.87it/s]
 84%|████████▎ | 8360/10000 [56:37<05:24,  5.06it/s]
                                                    
{'loss': 0.0607, 'grad_norm': 0.6681963205337524, 'learning_rate': 7.183324731875551e-06}

 84%|████████▎ | 8360/10000 [56:37<05:24,  5.06it/s]
 84%|████████▎ | 8361/10000 [56:38<05:41,  4.80it/s]
 84%|████████▎ | 8362/10000 [56:38<05:38,  4.84it/s]
 84%|████████▎ | 8363/10000 [56:38<05:42,  4.79it/s]
 84%|████████▎ | 8364/10000 [56:38<05:42,  4.77it/s]
 84%|████████▎ | 8365/10000 [56:39<05:43,  4.76it/s]
 84%|████████▎ | 8366/10000 [56:39<05:44,  4.74it/s]
 84%|████████▎ | 8367/10000 [56:39<05:39,  4.80it/s]
 84%|████████▎ | 8368/10000 [56:39<05:28,  4.96it/s]
 84%|████████▎ | 8369/10000 [56:39<05:20,  5.09it/s]
 84%|████████▎ | 8370/10000 [56:39<05:11,  5.23it/s]
                                                    
{'loss': 0.0669, 'grad_norm': 0.7523913383483887, 'learning_rate': 7.098169987248782e-06}

 84%|████████▎ | 8370/10000 [56:40<05:11,  5.23it/s]
 84%|████████▎ | 8371/10000 [56:40<05:16,  5.14it/s]
 84%|████████▎ | 8372/10000 [56:40<05:23,  5.03it/s]
 84%|████████▎ | 8373/10000 [56:40<05:30,  4.93it/s]
 84%|████████▎ | 8374/10000 [56:40<05:28,  4.95it/s]
 84%|████████▍ | 8375/10000 [56:40<05:24,  5.01it/s]
 84%|████████▍ | 8376/10000 [56:41<05:17,  5.12it/s]
 84%|████████▍ | 8377/10000 [56:41<05:06,  5.30it/s]
 84%|████████▍ | 8378/10000 [56:41<05:02,  5.37it/s]
 84%|████████▍ | 8379/10000 [56:41<04:58,  5.44it/s]
 84%|████████▍ | 8380/10000 [56:41<05:03,  5.33it/s]
                                                    
{'loss': 0.0559, 'grad_norm': 0.72547447681427, 'learning_rate': 7.013484410161553e-06}

 84%|████████▍ | 8380/10000 [56:41<05:03,  5.33it/s]
 84%|████████▍ | 8381/10000 [56:42<05:11,  5.20it/s]
 84%|████████▍ | 8382/10000 [56:42<05:08,  5.25it/s]
 84%|████████▍ | 8383/10000 [56:42<05:05,  5.29it/s]
 84%|████████▍ | 8384/10000 [56:42<05:02,  5.34it/s]
 84%|████████▍ | 8385/10000 [56:42<04:55,  5.46it/s]
 84%|████████▍ | 8386/10000 [56:43<04:51,  5.53it/s]
 84%|████████▍ | 8387/10000 [56:43<04:52,  5.52it/s]
 84%|████████▍ | 8388/10000 [56:43<04:52,  5.51it/s]
 84%|████████▍ | 8389/10000 [56:43<04:55,  5.45it/s]
 84%|████████▍ | 8390/10000 [56:43<05:04,  5.28it/s]
                                                    
{'loss': 0.0882, 'grad_norm': 0.5443186163902283, 'learning_rate': 6.92926892672176e-06}

 84%|████████▍ | 8390/10000 [56:43<05:04,  5.28it/s]
 84%|████████▍ | 8391/10000 [56:43<05:08,  5.22it/s]
 84%|████████▍ | 8392/10000 [56:44<05:05,  5.27it/s]
 84%|████████▍ | 8393/10000 [56:44<04:59,  5.37it/s]
 84%|████████▍ | 8394/10000 [56:44<04:54,  5.45it/s]
 84%|████████▍ | 8395/10000 [56:44<04:54,  5.45it/s]
 84%|████████▍ | 8396/10000 [56:44<05:02,  5.31it/s]
 84%|████████▍ | 8397/10000 [56:45<05:01,  5.32it/s]
 84%|████████▍ | 8398/10000 [56:45<04:57,  5.38it/s]
 84%|████████▍ | 8399/10000 [56:45<04:54,  5.43it/s]
 84%|████████▍ | 8400/10000 [56:45<04:55,  5.41it/s]
                                                    
{'loss': 0.0742, 'grad_norm': 0.5197641849517822, 'learning_rate': 6.845524457896446e-06}

 84%|████████▍ | 8400/10000 [56:45<04:55,  5.41it/s]
 84%|████████▍ | 8401/10000 [56:45<05:17,  5.03it/s]
 84%|████████▍ | 8402/10000 [56:46<05:37,  4.74it/s]
 84%|████████▍ | 8403/10000 [56:46<05:46,  4.61it/s]
 84%|████████▍ | 8404/10000 [56:46<05:50,  4.56it/s]
 84%|████████▍ | 8405/10000 [56:46<06:01,  4.42it/s]
 84%|████████▍ | 8406/10000 [56:46<05:41,  4.67it/s]
 84%|████████▍ | 8407/10000 [56:47<05:23,  4.93it/s]
 84%|████████▍ | 8408/10000 [56:47<05:10,  5.13it/s]
 84%|████████▍ | 8409/10000 [56:47<05:01,  5.27it/s]
 84%|████████▍ | 8410/10000 [56:47<05:06,  5.18it/s]
                                                    
{'loss': 0.0625, 'grad_norm': 0.7259860038757324, 'learning_rate': 6.7622519195017165e-06}

 84%|████████▍ | 8410/10000 [56:47<05:06,  5.18it/s]
 84%|████████▍ | 8411/10000 [56:47<05:33,  4.76it/s]
 84%|████████▍ | 8412/10000 [56:48<05:44,  4.61it/s]
 84%|████████▍ | 8413/10000 [56:48<05:50,  4.53it/s]
 84%|████████▍ | 8414/10000 [56:48<05:44,  4.60it/s]
 84%|████████▍ | 8415/10000 [56:48<05:28,  4.82it/s]
 84%|████████▍ | 8416/10000 [56:48<05:13,  5.05it/s]
 84%|████████▍ | 8417/10000 [56:49<05:02,  5.23it/s]
 84%|████████▍ | 8418/10000 [56:49<04:58,  5.29it/s]
 84%|████████▍ | 8419/10000 [56:49<05:00,  5.26it/s]
 84%|████████▍ | 8420/10000 [56:49<05:14,  5.03it/s]
                                                    
{'loss': 0.0584, 'grad_norm': 0.6402581930160522, 'learning_rate': 6.679452222192684e-06}

 84%|████████▍ | 8420/10000 [56:49<05:14,  5.03it/s]
 84%|████████▍ | 8421/10000 [56:49<05:27,  4.83it/s]
 84%|████████▍ | 8422/10000 [56:50<05:37,  4.67it/s]
 84%|████████▍ | 8423/10000 [56:50<05:32,  4.74it/s]
 84%|████████▍ | 8424/10000 [56:50<05:31,  4.76it/s]
 84%|████████▍ | 8425/10000 [56:50<05:15,  4.99it/s]
 84%|████████▍ | 8426/10000 [56:50<05:03,  5.19it/s]
 84%|████████▍ | 8427/10000 [56:51<04:55,  5.33it/s]
 84%|████████▍ | 8428/10000 [56:51<04:57,  5.28it/s]
 84%|████████▍ | 8429/10000 [56:51<05:06,  5.12it/s]
 84%|████████▍ | 8430/10000 [56:51<05:21,  4.88it/s]
                                                    
{'loss': 0.0695, 'grad_norm': 0.5328293442726135, 'learning_rate': 6.597126271453579e-06}

 84%|████████▍ | 8430/10000 [56:51<05:21,  4.88it/s]
 84%|████████▍ | 8431/10000 [56:52<05:32,  4.72it/s]
 84%|████████▍ | 8432/10000 [56:52<05:34,  4.69it/s]
 84%|████████▍ | 8433/10000 [56:52<05:31,  4.73it/s]
 84%|████████▍ | 8434/10000 [56:52<05:14,  4.99it/s]
 84%|████████▍ | 8435/10000 [56:52<05:02,  5.17it/s]
 84%|████████▍ | 8436/10000 [56:52<04:58,  5.23it/s]
 84%|████████▍ | 8437/10000 [56:53<05:10,  5.03it/s]
 84%|████████▍ | 8438/10000 [56:53<05:19,  4.88it/s]
 84%|████████▍ | 8439/10000 [56:53<05:34,  4.67it/s]
 84%|████████▍ | 8440/10000 [56:53<05:38,  4.60it/s]
                                                    
{'loss': 0.0654, 'grad_norm': 0.6357295513153076, 'learning_rate': 6.51527496758782e-06}

 84%|████████▍ | 8440/10000 [56:53<05:38,  4.60it/s]
 84%|████████▍ | 8441/10000 [56:54<05:35,  4.64it/s]
 84%|████████▍ | 8442/10000 [56:54<05:20,  4.86it/s]
 84%|████████▍ | 8443/10000 [56:54<05:06,  5.07it/s]
 84%|████████▍ | 8444/10000 [56:54<05:06,  5.07it/s]
 84%|████████▍ | 8445/10000 [56:54<05:12,  4.98it/s]
 84%|████████▍ | 8446/10000 [56:55<05:18,  4.88it/s]
 84%|████████▍ | 8447/10000 [56:55<05:27,  4.74it/s]
 84%|████████▍ | 8448/10000 [56:55<05:39,  4.58it/s]
 84%|████████▍ | 8449/10000 [56:55<05:46,  4.47it/s]
 84%|████████▍ | 8450/10000 [56:55<05:47,  4.46it/s]
                                                    
{'loss': 0.0688, 'grad_norm': 0.5662961602210999, 'learning_rate': 6.433899205708155e-06}

 84%|████████▍ | 8450/10000 [56:56<05:47,  4.46it/s]
 85%|████████▍ | 8451/10000 [56:56<05:45,  4.48it/s]Rank 0, Worker 3: Wait for shard 41 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 85%|████████▍ | 8452/10000 [56:56<05:51,  4.40it/s]
 85%|████████▍ | 8453/10000 [56:56<05:43,  4.50it/s]
 85%|████████▍ | 8454/10000 [56:56<05:47,  4.45it/s]
 85%|████████▍ | 8455/10000 [56:57<06:01,  4.27it/s]
 85%|████████▍ | 8456/10000 [56:57<06:19,  4.07it/s]
 85%|████████▍ | 8457/10000 [56:57<05:57,  4.32it/s]
 85%|████████▍ | 8458/10000 [56:57<05:40,  4.53it/s]
 85%|████████▍ | 8459/10000 [56:58<05:33,  4.62it/s]
 85%|████████▍ | 8460/10000 [56:58<05:39,  4.53it/s]
                                                    
{'loss': 0.0566, 'grad_norm': 0.5495201945304871, 'learning_rate': 6.352999875726856e-06}

 85%|████████▍ | 8460/10000 [56:58<05:39,  4.53it/s]
 85%|████████▍ | 8461/10000 [56:58<05:49,  4.40it/s]
 85%|████████▍ | 8462/10000 [56:58<05:53,  4.36it/s]
 85%|████████▍ | 8463/10000 [56:58<05:46,  4.43it/s]
 85%|████████▍ | 8464/10000 [56:59<05:37,  4.55it/s]
 85%|████████▍ | 8465/10000 [56:59<05:30,  4.64it/s]
 85%|████████▍ | 8466/10000 [56:59<05:26,  4.70it/s]
 85%|████████▍ | 8467/10000 [56:59<05:24,  4.72it/s]
 85%|████████▍ | 8468/10000 [57:00<05:36,  4.55it/s]
 85%|████████▍ | 8469/10000 [57:00<05:32,  4.61it/s]
 85%|████████▍ | 8470/10000 [57:00<05:28,  4.66it/s]
                                                    
{'loss': 0.0585, 'grad_norm': 0.5902302861213684, 'learning_rate': 6.272577862346052e-06}

 85%|████████▍ | 8470/10000 [57:00<05:28,  4.66it/s]
 85%|████████▍ | 8471/10000 [57:00<05:31,  4.62it/s]Rank 0, Worker 5: Wait for shard 69 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 85%|████████▍ | 8472/10000 [57:00<05:46,  4.41it/s]
 85%|████████▍ | 8473/10000 [57:01<05:48,  4.38it/s]
 85%|████████▍ | 8474/10000 [57:01<05:44,  4.43it/s]
 85%|████████▍ | 8475/10000 [57:01<05:44,  4.43it/s]
 85%|████████▍ | 8476/10000 [57:01<05:38,  4.50it/s]
 85%|████████▍ | 8477/10000 [57:02<05:40,  4.48it/s]
 85%|████████▍ | 8478/10000 [57:02<05:50,  4.34it/s]
 85%|████████▍ | 8479/10000 [57:02<05:59,  4.24it/s]
 85%|████████▍ | 8480/10000 [57:02<05:58,  4.24it/s]
                                                    
{'loss': 0.0909, 'grad_norm': 0.5881245136260986, 'learning_rate': 6.192634045047996e-06}

 85%|████████▍ | 8480/10000 [57:02<05:58,  4.24it/s]
 85%|████████▍ | 8481/10000 [57:03<06:25,  3.94it/s]
 85%|████████▍ | 8482/10000 [57:03<06:19,  4.00it/s]
 85%|████████▍ | 8483/10000 [57:03<06:11,  4.09it/s]
 85%|████████▍ | 8484/10000 [57:03<05:57,  4.24it/s]
 85%|████████▍ | 8485/10000 [57:03<05:42,  4.43it/s]
 85%|████████▍ | 8486/10000 [57:04<05:33,  4.54it/s]
 85%|████████▍ | 8487/10000 [57:04<05:24,  4.66it/s]
 85%|████████▍ | 8488/10000 [57:04<05:33,  4.53it/s]
 85%|████████▍ | 8489/10000 [57:04<05:42,  4.41it/s]
 85%|████████▍ | 8490/10000 [57:05<06:01,  4.18it/s]
                                                    
{'loss': 0.0693, 'grad_norm': 0.6763169765472412, 'learning_rate': 6.113169298085458e-06}

 85%|████████▍ | 8490/10000 [57:05<06:01,  4.18it/s]
 85%|████████▍ | 8491/10000 [57:05<06:14,  4.03it/s]
 85%|████████▍ | 8492/10000 [57:05<06:09,  4.08it/s]
 85%|████████▍ | 8493/10000 [57:05<06:03,  4.14it/s]
 85%|████████▍ | 8494/10000 [57:06<05:45,  4.36it/s]
 85%|████████▍ | 8495/10000 [57:06<05:39,  4.44it/s]
 85%|████████▍ | 8496/10000 [57:06<05:39,  4.43it/s]
 85%|████████▍ | 8497/10000 [57:06<05:44,  4.36it/s]
 85%|████████▍ | 8498/10000 [57:06<05:50,  4.29it/s]
 85%|████████▍ | 8499/10000 [57:07<05:54,  4.23it/s]
 85%|████████▌ | 8500/10000 [57:07<05:55,  4.21it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 17.03 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 18.35 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 18.46 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 18.64 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 18.71 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 18.84 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 8500: eval/loss = 0.0903 (36s)

                                                    
{'loss': 0.0751, 'grad_norm': 0.582548201084137, 'learning_rate': 6.034184490472195e-06}

 85%|████████▌ | 8500/10000 [57:43<05:55,  4.21it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-8500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-8500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-8500/wandb_config.json

 85%|████████▌ | 8501/10000 [58:10<8:00:21, 19.23s/it]
 85%|████████▌ | 8502/10000 [58:11<5:37:37, 13.52s/it]Rank 0, Worker 0: Wait for shard 19 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 85%|████████▌ | 8503/10000 [58:11<3:57:51,  9.53s/it]
 85%|████████▌ | 8504/10000 [58:11<2:48:04,  6.74s/it]
 85%|████████▌ | 8505/10000 [58:11<1:59:07,  4.78s/it]
 85%|████████▌ | 8506/10000 [58:12<1:24:56,  3.41s/it]
 85%|████████▌ | 8507/10000 [58:12<1:00:50,  2.44s/it]
 85%|████████▌ | 8508/10000 [58:12<44:03,  1.77s/it]  
 85%|████████▌ | 8509/10000 [58:12<32:15,  1.30s/it]
 85%|████████▌ | 8510/10000 [58:12<23:59,  1.04it/s]
                                                    
{'loss': 0.0751, 'grad_norm': 0.5758833289146423, 'learning_rate': 5.955680485973386e-06}

 85%|████████▌ | 8510/10000 [58:12<23:59,  1.04it/s]
 85%|████████▌ | 8511/10000 [58:13<18:32,  1.34it/s]
 85%|████████▌ | 8512/10000 [58:13<14:35,  1.70it/s]
 85%|████████▌ | 8513/10000 [58:13<11:44,  2.11it/s]
 85%|████████▌ | 8514/10000 [58:13<09:50,  2.52it/s]
 85%|████████▌ | 8515/10000 [58:13<08:26,  2.93it/s]
 85%|████████▌ | 8516/10000 [58:14<07:18,  3.38it/s]
 85%|████████▌ | 8517/10000 [58:14<06:34,  3.75it/s]
 85%|████████▌ | 8518/10000 [58:14<06:05,  4.06it/s]
 85%|████████▌ | 8519/10000 [58:14<05:43,  4.31it/s]
 85%|████████▌ | 8520/10000 [58:14<05:29,  4.49it/s]
                                                    
{'loss': 0.0628, 'grad_norm': 0.5767698884010315, 'learning_rate': 5.877658143096265e-06}

 85%|████████▌ | 8520/10000 [58:14<05:29,  4.49it/s]
 85%|████████▌ | 8521/10000 [58:15<05:29,  4.49it/s]
 85%|████████▌ | 8522/10000 [58:15<05:26,  4.53it/s]
 85%|████████▌ | 8523/10000 [58:15<05:24,  4.55it/s]
 85%|████████▌ | 8524/10000 [58:15<05:20,  4.61it/s]
 85%|████████▌ | 8525/10000 [58:15<05:17,  4.65it/s]
 85%|████████▌ | 8526/10000 [58:16<05:08,  4.79it/s]
 85%|████████▌ | 8527/10000 [58:16<05:01,  4.89it/s]
 85%|████████▌ | 8528/10000 [58:16<04:55,  4.98it/s]
 85%|████████▌ | 8529/10000 [58:16<04:56,  4.96it/s]
 85%|████████▌ | 8530/10000 [58:16<05:01,  4.88it/s]
                                                    
{'loss': 0.0714, 'grad_norm': 0.5791407823562622, 'learning_rate': 5.800118315080661e-06}

 85%|████████▌ | 8530/10000 [58:17<05:01,  4.88it/s]
 85%|████████▌ | 8531/10000 [58:17<05:10,  4.73it/s]
 85%|████████▌ | 8532/10000 [58:17<05:09,  4.74it/s]
 85%|████████▌ | 8533/10000 [58:17<05:06,  4.78it/s]
 85%|████████▌ | 8534/10000 [58:17<05:05,  4.79it/s]
 85%|████████▌ | 8535/10000 [58:18<05:10,  4.71it/s]
 85%|████████▌ | 8536/10000 [58:18<05:02,  4.83it/s]
 85%|████████▌ | 8537/10000 [58:18<04:53,  4.98it/s]
 85%|████████▌ | 8538/10000 [58:18<04:51,  5.01it/s]
 85%|████████▌ | 8539/10000 [58:18<04:56,  4.92it/s]
 85%|████████▌ | 8540/10000 [58:19<05:01,  4.83it/s]
                                                    
{'loss': 0.0732, 'grad_norm': 1.1255528926849365, 'learning_rate': 5.723061849889716e-06}

 85%|████████▌ | 8540/10000 [58:19<05:01,  4.83it/s]
 85%|████████▌ | 8541/10000 [58:19<05:20,  4.55it/s]
 85%|████████▌ | 8542/10000 [58:19<05:24,  4.49it/s]
 85%|████████▌ | 8543/10000 [58:19<05:21,  4.53it/s]
 85%|████████▌ | 8544/10000 [58:19<05:09,  4.70it/s]
 85%|████████▌ | 8545/10000 [58:20<05:01,  4.83it/s]
 85%|████████▌ | 8546/10000 [58:20<04:59,  4.85it/s]
 85%|████████▌ | 8547/10000 [58:20<05:02,  4.81it/s]
 85%|████████▌ | 8548/10000 [58:20<05:05,  4.75it/s]
 85%|████████▌ | 8549/10000 [58:20<05:09,  4.69it/s]
 86%|████████▌ | 8550/10000 [58:21<05:07,  4.72it/s]
                                                    
{'loss': 0.0577, 'grad_norm': 0.6093800663948059, 'learning_rate': 5.646489590200604e-06}

 86%|████████▌ | 8550/10000 [58:21<05:07,  4.72it/s]
 86%|████████▌ | 8551/10000 [58:21<05:05,  4.74it/s]
 86%|████████▌ | 8552/10000 [58:21<04:58,  4.84it/s]
 86%|████████▌ | 8553/10000 [58:21<04:52,  4.95it/s]
 86%|████████▌ | 8554/10000 [58:22<05:19,  4.52it/s]
 86%|████████▌ | 8555/10000 [58:22<05:10,  4.65it/s]
 86%|████████▌ | 8556/10000 [58:22<05:07,  4.70it/s]
 86%|████████▌ | 8557/10000 [58:22<05:10,  4.65it/s]
 86%|████████▌ | 8558/10000 [58:22<05:06,  4.70it/s]
 86%|████████▌ | 8559/10000 [58:23<05:16,  4.56it/s]
 86%|████████▌ | 8560/10000 [58:23<05:12,  4.61it/s]
                                                    
{'loss': 0.0688, 'grad_norm': 0.6241508722305298, 'learning_rate': 5.570402373395256e-06}

 86%|████████▌ | 8560/10000 [58:23<05:12,  4.61it/s]
 86%|████████▌ | 8561/10000 [58:23<05:13,  4.59it/s]
 86%|████████▌ | 8562/10000 [58:23<05:09,  4.64it/s]
 86%|████████▌ | 8563/10000 [58:23<05:10,  4.63it/s]
 86%|████████▌ | 8564/10000 [58:24<05:09,  4.64it/s]
 86%|████████▌ | 8565/10000 [58:24<05:13,  4.58it/s]
 86%|████████▌ | 8566/10000 [58:24<05:10,  4.62it/s]
 86%|████████▌ | 8567/10000 [58:24<05:10,  4.62it/s]
 86%|████████▌ | 8568/10000 [58:25<05:07,  4.66it/s]
 86%|████████▌ | 8569/10000 [58:25<04:59,  4.79it/s]
 86%|████████▌ | 8570/10000 [58:25<04:55,  4.85it/s]
                                                    
{'loss': 0.0719, 'grad_norm': 0.5808899402618408, 'learning_rate': 5.494801031551305e-06}

 86%|████████▌ | 8570/10000 [58:25<04:55,  4.85it/s]
 86%|████████▌ | 8571/10000 [58:25<04:58,  4.79it/s]
 86%|████████▌ | 8572/10000 [58:25<04:55,  4.83it/s]
 86%|████████▌ | 8573/10000 [58:26<05:05,  4.67it/s]
 86%|████████▌ | 8574/10000 [58:26<05:02,  4.71it/s]
 86%|████████▌ | 8575/10000 [58:26<05:02,  4.72it/s]
 86%|████████▌ | 8576/10000 [58:26<05:02,  4.70it/s]
 86%|████████▌ | 8577/10000 [58:26<05:00,  4.73it/s]
 86%|████████▌ | 8578/10000 [58:27<05:01,  4.71it/s]
 86%|████████▌ | 8579/10000 [58:27<05:02,  4.70it/s]
 86%|████████▌ | 8580/10000 [58:27<04:56,  4.79it/s]
                                                    
{'loss': 0.0865, 'grad_norm': 0.6516720056533813, 'learning_rate': 5.41968639143291e-06}

 86%|████████▌ | 8580/10000 [58:27<04:56,  4.79it/s]
 86%|████████▌ | 8581/10000 [58:27<05:12,  4.54it/s]
 86%|████████▌ | 8582/10000 [58:28<05:10,  4.57it/s]
 86%|████████▌ | 8583/10000 [58:28<05:13,  4.52it/s]
 86%|████████▌ | 8584/10000 [58:28<05:20,  4.42it/s]
 86%|████████▌ | 8585/10000 [58:28<05:14,  4.50it/s]
 86%|████████▌ | 8586/10000 [58:28<05:19,  4.42it/s]
 86%|████████▌ | 8587/10000 [58:29<05:24,  4.35it/s]
 86%|████████▌ | 8588/10000 [58:29<05:20,  4.40it/s]
 86%|████████▌ | 8589/10000 [58:29<05:14,  4.49it/s]
 86%|████████▌ | 8590/10000 [58:29<05:06,  4.61it/s]
                                                    
{'loss': 0.0589, 'grad_norm': 0.6727027297019958, 'learning_rate': 5.345059274481751e-06}

 86%|████████▌ | 8590/10000 [58:29<05:06,  4.61it/s]
 86%|████████▌ | 8591/10000 [58:30<05:08,  4.57it/s]
 86%|████████▌ | 8592/10000 [58:30<05:02,  4.65it/s]
 86%|████████▌ | 8593/10000 [58:30<04:59,  4.69it/s]
 86%|████████▌ | 8594/10000 [58:30<05:11,  4.51it/s]
 86%|████████▌ | 8595/10000 [58:30<05:22,  4.36it/s]
 86%|████████▌ | 8596/10000 [58:31<05:29,  4.27it/s]
 86%|████████▌ | 8597/10000 [58:31<05:28,  4.28it/s]
 86%|████████▌ | 8598/10000 [58:31<05:23,  4.34it/s]
 86%|████████▌ | 8599/10000 [58:31<05:19,  4.38it/s]
 86%|████████▌ | 8600/10000 [58:32<05:15,  4.44it/s]
                                                    
{'loss': 0.0689, 'grad_norm': 0.635355532169342, 'learning_rate': 5.270920496808002e-06}

 86%|████████▌ | 8600/10000 [58:32<05:15,  4.44it/s]
 86%|████████▌ | 8601/10000 [58:32<05:20,  4.36it/s]
 86%|████████▌ | 8602/10000 [58:32<05:28,  4.25it/s]
 86%|████████▌ | 8603/10000 [58:32<05:40,  4.11it/s]
 86%|████████▌ | 8604/10000 [58:33<05:47,  4.02it/s]
 86%|████████▌ | 8605/10000 [58:33<05:38,  4.12it/s]
 86%|████████▌ | 8606/10000 [58:33<05:26,  4.27it/s]
 86%|████████▌ | 8607/10000 [58:33<05:13,  4.45it/s]
 86%|████████▌ | 8608/10000 [58:33<05:01,  4.62it/s]
 86%|████████▌ | 8609/10000 [58:34<04:54,  4.73it/s]
 86%|████████▌ | 8610/10000 [58:34<04:55,  4.70it/s]
                                                    
{'loss': 0.0724, 'grad_norm': 0.6256790161132812, 'learning_rate': 5.1972708691814695e-06}

 86%|████████▌ | 8610/10000 [58:34<04:55,  4.70it/s]
 86%|████████▌ | 8611/10000 [58:34<04:59,  4.64it/s]
 86%|████████▌ | 8612/10000 [58:34<04:59,  4.64it/s]
 86%|████████▌ | 8613/10000 [58:35<05:09,  4.48it/s]
 86%|████████▌ | 8614/10000 [58:35<05:11,  4.44it/s]
 86%|████████▌ | 8615/10000 [58:35<05:10,  4.46it/s]
 86%|████████▌ | 8616/10000 [58:35<05:05,  4.53it/s]
 86%|████████▌ | 8617/10000 [58:35<04:55,  4.67it/s]
 86%|████████▌ | 8618/10000 [58:36<04:54,  4.69it/s]
 86%|████████▌ | 8619/10000 [58:36<04:55,  4.68it/s]
 86%|████████▌ | 8620/10000 [58:36<04:49,  4.76it/s]
                                                    
{'loss': 0.067, 'grad_norm': 1.0519378185272217, 'learning_rate': 5.124111197022674e-06}

 86%|████████▌ | 8620/10000 [58:36<04:49,  4.76it/s]
 86%|████████▌ | 8621/10000 [58:36<04:53,  4.70it/s]
 86%|████████▌ | 8622/10000 [58:37<05:10,  4.44it/s]
 86%|████████▌ | 8623/10000 [58:37<05:13,  4.40it/s]
 86%|████████▌ | 8624/10000 [58:37<05:00,  4.57it/s]
 86%|████████▋ | 8625/10000 [58:37<04:53,  4.69it/s]
 86%|████████▋ | 8626/10000 [58:37<04:46,  4.80it/s]
 86%|████████▋ | 8627/10000 [58:38<04:44,  4.83it/s]
 86%|████████▋ | 8628/10000 [58:38<04:46,  4.80it/s]
 86%|████████▋ | 8629/10000 [58:38<04:44,  4.82it/s]
 86%|████████▋ | 8630/10000 [58:38<04:42,  4.84it/s]
                                                    
{'loss': 0.0672, 'grad_norm': 0.7039864659309387, 'learning_rate': 5.051442280394081e-06}

 86%|████████▋ | 8630/10000 [58:38<04:42,  4.84it/s]
 86%|████████▋ | 8631/10000 [58:38<04:50,  4.71it/s]
 86%|████████▋ | 8632/10000 [58:39<04:54,  4.65it/s]
 86%|████████▋ | 8633/10000 [58:39<04:48,  4.73it/s]
 86%|████████▋ | 8634/10000 [58:39<04:41,  4.86it/s]
 86%|████████▋ | 8635/10000 [58:39<04:36,  4.94it/s]
 86%|████████▋ | 8636/10000 [58:39<04:34,  4.97it/s]
 86%|████████▋ | 8637/10000 [58:40<04:35,  4.95it/s]
 86%|████████▋ | 8638/10000 [58:40<04:45,  4.78it/s]
 86%|████████▋ | 8639/10000 [58:40<04:42,  4.81it/s]
 86%|████████▋ | 8640/10000 [58:40<04:41,  4.83it/s]
                                                    
{'loss': 0.0806, 'grad_norm': 0.595604419708252, 'learning_rate': 4.979264913991322e-06}

 86%|████████▋ | 8640/10000 [58:40<04:41,  4.83it/s]
 86%|████████▋ | 8641/10000 [58:40<04:39,  4.86it/s]
 86%|████████▋ | 8642/10000 [58:41<04:27,  5.08it/s]
 86%|████████▋ | 8643/10000 [58:41<04:24,  5.13it/s]
 86%|████████▋ | 8644/10000 [58:41<04:23,  5.15it/s]
 86%|████████▋ | 8645/10000 [58:41<04:26,  5.09it/s]
 86%|████████▋ | 8646/10000 [58:41<04:34,  4.94it/s]
 86%|████████▋ | 8647/10000 [58:42<04:34,  4.93it/s]
 86%|████████▋ | 8648/10000 [58:42<04:34,  4.93it/s]
 86%|████████▋ | 8649/10000 [58:42<04:32,  4.95it/s]
 86%|████████▋ | 8650/10000 [58:42<04:29,  5.00it/s]
                                                    
{'loss': 0.0909, 'grad_norm': 0.7049185633659363, 'learning_rate': 4.907579887134489e-06}

 86%|████████▋ | 8650/10000 [58:42<04:29,  5.00it/s]
 87%|████████▋ | 8651/10000 [58:42<04:34,  4.92it/s]
 87%|████████▋ | 8652/10000 [58:43<04:30,  4.99it/s]
 87%|████████▋ | 8653/10000 [58:43<04:28,  5.01it/s]
 87%|████████▋ | 8654/10000 [58:43<04:30,  4.97it/s]
 87%|████████▋ | 8655/10000 [58:43<04:31,  4.95it/s]
 87%|████████▋ | 8656/10000 [58:43<04:31,  4.94it/s]
 87%|████████▋ | 8657/10000 [58:44<04:34,  4.90it/s]
 87%|████████▋ | 8658/10000 [58:44<04:34,  4.89it/s]
 87%|████████▋ | 8659/10000 [58:44<04:33,  4.90it/s]
 87%|████████▋ | 8660/10000 [58:44<04:27,  5.01it/s]
                                                    
{'loss': 0.0552, 'grad_norm': 0.5481497049331665, 'learning_rate': 4.836387983759572e-06}

 87%|████████▋ | 8660/10000 [58:44<04:27,  5.01it/s]
 87%|████████▋ | 8661/10000 [58:44<04:27,  5.00it/s]
 87%|████████▋ | 8662/10000 [58:45<04:22,  5.10it/s]
 87%|████████▋ | 8663/10000 [58:45<04:13,  5.28it/s]
 87%|████████▋ | 8664/10000 [58:45<04:07,  5.39it/s]
 87%|████████▋ | 8665/10000 [58:45<04:02,  5.50it/s]
 87%|████████▋ | 8666/10000 [58:45<04:01,  5.53it/s]
 87%|████████▋ | 8667/10000 [58:46<03:58,  5.58it/s]
 87%|████████▋ | 8668/10000 [58:46<03:57,  5.61it/s]
 87%|████████▋ | 8669/10000 [58:46<03:56,  5.62it/s]
 87%|████████▋ | 8670/10000 [58:46<03:55,  5.64it/s]
                                                    
{'loss': 0.0535, 'grad_norm': 0.6109694838523865, 'learning_rate': 4.765689982409816e-06}

 87%|████████▋ | 8670/10000 [58:46<03:55,  5.64it/s]
 87%|████████▋ | 8671/10000 [58:46<03:57,  5.59it/s]
 87%|████████▋ | 8672/10000 [58:46<03:55,  5.64it/s]
 87%|████████▋ | 8673/10000 [58:47<03:55,  5.65it/s]
 87%|████████▋ | 8674/10000 [58:47<04:01,  5.48it/s]
 87%|████████▋ | 8675/10000 [58:47<04:04,  5.42it/s]
 87%|████████▋ | 8676/10000 [58:47<04:06,  5.37it/s]
 87%|████████▋ | 8677/10000 [58:47<04:10,  5.27it/s]
 87%|████████▋ | 8678/10000 [58:48<04:09,  5.30it/s]
 87%|████████▋ | 8679/10000 [58:48<04:10,  5.27it/s]
 87%|████████▋ | 8680/10000 [58:48<04:11,  5.26it/s]
                                                    
{'loss': 0.075, 'grad_norm': 0.613645613193512, 'learning_rate': 4.695486656227233e-06}

 87%|████████▋ | 8680/10000 [58:48<04:11,  5.26it/s]
 87%|████████▋ | 8681/10000 [58:48<04:15,  5.17it/s]
 87%|████████▋ | 8682/10000 [58:48<04:16,  5.14it/s]
 87%|████████▋ | 8683/10000 [58:48<04:11,  5.25it/s]
 87%|████████▋ | 8684/10000 [58:49<04:05,  5.35it/s]
 87%|████████▋ | 8685/10000 [58:49<04:07,  5.32it/s]
 87%|████████▋ | 8686/10000 [58:49<04:03,  5.39it/s]
 87%|████████▋ | 8687/10000 [58:49<04:09,  5.27it/s]
 87%|████████▋ | 8688/10000 [58:49<04:09,  5.25it/s]
 87%|████████▋ | 8689/10000 [58:50<04:08,  5.28it/s]
 87%|████████▋ | 8690/10000 [58:50<04:11,  5.21it/s]
                                                    
{'loss': 0.0628, 'grad_norm': 0.6890705823898315, 'learning_rate': 4.625778772944156e-06}

 87%|████████▋ | 8690/10000 [58:50<04:11,  5.21it/s]
 87%|████████▋ | 8691/10000 [58:50<04:11,  5.20it/s]
 87%|████████▋ | 8692/10000 [58:50<04:07,  5.29it/s]
 87%|████████▋ | 8693/10000 [58:50<04:03,  5.37it/s]
 87%|████████▋ | 8694/10000 [58:51<04:05,  5.32it/s]
 87%|████████▋ | 8695/10000 [58:51<04:10,  5.21it/s]
 87%|████████▋ | 8696/10000 [58:51<04:13,  5.14it/s]
 87%|████████▋ | 8697/10000 [58:51<04:15,  5.11it/s]
 87%|████████▋ | 8698/10000 [58:51<04:12,  5.16it/s]
 87%|████████▋ | 8699/10000 [58:52<04:10,  5.20it/s]
 87%|████████▋ | 8700/10000 [58:52<04:08,  5.24it/s]
                                                    
{'loss': 0.0632, 'grad_norm': 0.5794233679771423, 'learning_rate': 4.556567094874825e-06}

 87%|████████▋ | 8700/10000 [58:52<04:08,  5.24it/s]
 87%|████████▋ | 8701/10000 [58:52<04:07,  5.25it/s]
 87%|████████▋ | 8702/10000 [58:52<04:07,  5.23it/s]
 87%|████████▋ | 8703/10000 [58:52<04:12,  5.13it/s]
 87%|████████▋ | 8704/10000 [58:53<04:13,  5.12it/s]
 87%|████████▋ | 8705/10000 [58:53<04:16,  5.05it/s]
 87%|████████▋ | 8706/10000 [58:53<04:17,  5.03it/s]
 87%|████████▋ | 8707/10000 [58:53<04:15,  5.05it/s]
 87%|████████▋ | 8708/10000 [58:53<04:17,  5.01it/s]
 87%|████████▋ | 8709/10000 [58:54<04:09,  5.17it/s]
 87%|████████▋ | 8710/10000 [58:54<04:03,  5.30it/s]
                                                    
{'loss': 0.0615, 'grad_norm': 0.8483870625495911, 'learning_rate': 4.487852378907059e-06}

 87%|████████▋ | 8710/10000 [58:54<04:03,  5.30it/s]
 87%|████████▋ | 8711/10000 [58:54<04:03,  5.29it/s]
 87%|████████▋ | 8712/10000 [58:54<04:00,  5.35it/s]
 87%|████████▋ | 8713/10000 [58:54<04:03,  5.29it/s]
 87%|████████▋ | 8714/10000 [58:54<04:07,  5.20it/s]
 87%|████████▋ | 8715/10000 [58:55<04:08,  5.17it/s]
 87%|████████▋ | 8716/10000 [58:55<04:10,  5.12it/s]
 87%|████████▋ | 8717/10000 [58:55<04:13,  5.07it/s]
 87%|████████▋ | 8718/10000 [58:55<04:10,  5.11it/s]
 87%|████████▋ | 8719/10000 [58:55<04:05,  5.22it/s]
 87%|████████▋ | 8720/10000 [58:56<04:00,  5.32it/s]
                                                    
{'loss': 0.069, 'grad_norm': 0.7187392115592957, 'learning_rate': 4.419635376493986e-06}

 87%|████████▋ | 8720/10000 [58:56<04:00,  5.32it/s]
 87%|████████▋ | 8721/10000 [58:56<04:01,  5.30it/s]
 87%|████████▋ | 8722/10000 [58:56<04:01,  5.29it/s]
 87%|████████▋ | 8723/10000 [58:56<04:01,  5.28it/s]
 87%|████████▋ | 8724/10000 [58:56<04:04,  5.22it/s]
 87%|████████▋ | 8725/10000 [58:57<04:08,  5.14it/s]
 87%|████████▋ | 8726/10000 [58:57<04:10,  5.08it/s]
 87%|████████▋ | 8727/10000 [58:57<04:10,  5.09it/s]
 87%|████████▋ | 8728/10000 [58:57<04:07,  5.15it/s]
 87%|████████▋ | 8729/10000 [58:57<04:05,  5.18it/s]
 87%|████████▋ | 8730/10000 [58:58<04:00,  5.29it/s]
                                                    
{'loss': 0.0515, 'grad_norm': 0.6460190415382385, 'learning_rate': 4.351916833645825e-06}

 87%|████████▋ | 8730/10000 [58:58<04:00,  5.29it/s]
 87%|████████▋ | 8731/10000 [58:58<04:03,  5.21it/s]
 87%|████████▋ | 8732/10000 [58:58<04:07,  5.13it/s]
 87%|████████▋ | 8733/10000 [58:58<04:06,  5.15it/s]
 87%|████████▋ | 8734/10000 [58:58<04:06,  5.13it/s]
 87%|████████▋ | 8735/10000 [58:59<04:06,  5.13it/s]
 87%|████████▋ | 8736/10000 [58:59<04:04,  5.16it/s]
 87%|████████▋ | 8737/10000 [58:59<04:05,  5.14it/s]
 87%|████████▋ | 8738/10000 [58:59<04:06,  5.12it/s]
 87%|████████▋ | 8739/10000 [58:59<04:05,  5.13it/s]
 87%|████████▋ | 8740/10000 [58:59<04:07,  5.08it/s]
                                                    
{'loss': 0.0645, 'grad_norm': 0.5117931365966797, 'learning_rate': 4.284697490921691e-06}

 87%|████████▋ | 8740/10000 [59:00<04:07,  5.08it/s]
 87%|████████▋ | 8741/10000 [59:00<04:09,  5.05it/s]
 87%|████████▋ | 8742/10000 [59:00<04:09,  5.04it/s]
 87%|████████▋ | 8743/10000 [59:00<04:14,  4.93it/s]
 87%|████████▋ | 8744/10000 [59:00<04:15,  4.91it/s]
 87%|████████▋ | 8745/10000 [59:01<04:22,  4.79it/s]
 87%|████████▋ | 8746/10000 [59:01<04:16,  4.88it/s]
 87%|████████▋ | 8747/10000 [59:01<04:14,  4.93it/s]
 87%|████████▋ | 8748/10000 [59:01<04:11,  4.98it/s]
 87%|████████▋ | 8749/10000 [59:01<04:07,  5.05it/s]
 88%|████████▊ | 8750/10000 [59:01<04:02,  5.16it/s]Rank 0, Worker 1: Caching shard...
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 14.66 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 15.95 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 16.18 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 16.21 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 16.25 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 16.30 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 8750: eval/loss = 0.0796 (34s)

                                                    
{'loss': 0.0826, 'grad_norm': 0.7660027742385864, 'learning_rate': 4.2179780834215585e-06}

 88%|████████▊ | 8750/10000 [59:36<04:02,  5.16it/s]
 88%|████████▊ | 8751/10000 [59:36<3:37:50, 10.46s/it]
 88%|████████▊ | 8752/10000 [59:36<2:33:31,  7.38s/it]
 88%|████████▊ | 8753/10000 [59:36<1:48:43,  5.23s/it]
 88%|████████▊ | 8754/10000 [59:37<1:17:16,  3.72s/it]
 88%|████████▊ | 8755/10000 [59:37<55:08,  2.66s/it]  
 88%|████████▊ | 8756/10000 [59:37<39:41,  1.91s/it]
 88%|████████▊ | 8757/10000 [59:37<28:51,  1.39s/it]
 88%|████████▊ | 8758/10000 [59:37<21:16,  1.03s/it]
 88%|████████▊ | 8759/10000 [59:37<15:59,  1.29it/s]
 88%|████████▊ | 8760/10000 [59:38<12:25,  1.66it/s]
                                                    
{'loss': 0.0765, 'grad_norm': 0.614324152469635, 'learning_rate': 4.151759340778178e-06}

 88%|████████▊ | 8760/10000 [59:38<12:25,  1.66it/s]
 88%|████████▊ | 8761/10000 [59:38<10:02,  2.06it/s]
 88%|████████▊ | 8762/10000 [59:38<08:12,  2.51it/s]
 88%|████████▊ | 8763/10000 [59:38<06:54,  2.98it/s]
 88%|████████▊ | 8764/10000 [59:38<05:55,  3.47it/s]
 88%|████████▊ | 8765/10000 [59:39<05:15,  3.92it/s]
 88%|████████▊ | 8766/10000 [59:39<04:55,  4.17it/s]
 88%|████████▊ | 8767/10000 [59:39<04:34,  4.49it/s]
 88%|████████▊ | 8768/10000 [59:39<04:18,  4.76it/s]
 88%|████████▊ | 8769/10000 [59:39<04:05,  5.01it/s]
 88%|████████▊ | 8770/10000 [59:39<04:01,  5.10it/s]
                                                    
{'loss': 0.0651, 'grad_norm': 0.7510131001472473, 'learning_rate': 4.086041987149109e-06}

 88%|████████▊ | 8770/10000 [59:40<04:01,  5.10it/s]
 88%|████████▊ | 8771/10000 [59:40<04:03,  5.05it/s]
 88%|████████▊ | 8772/10000 [59:40<04:11,  4.88it/s]
 88%|████████▊ | 8773/10000 [59:40<04:16,  4.78it/s]
 88%|████████▊ | 8774/10000 [59:40<04:19,  4.72it/s]
 88%|████████▊ | 8775/10000 [59:41<04:26,  4.60it/s]
 88%|████████▊ | 8776/10000 [59:41<04:26,  4.59it/s]
 88%|████████▊ | 8777/10000 [59:41<04:13,  4.83it/s]
 88%|████████▊ | 8778/10000 [59:41<04:01,  5.05it/s]
 88%|████████▊ | 8779/10000 [59:41<03:53,  5.23it/s]
 88%|████████▊ | 8780/10000 [59:42<03:48,  5.33it/s]
                                                    
{'loss': 0.0662, 'grad_norm': 0.5936854481697083, 'learning_rate': 4.020826741208811e-06}

 88%|████████▊ | 8780/10000 [59:42<03:48,  5.33it/s]
 88%|████████▊ | 8781/10000 [59:42<04:02,  5.02it/s]
 88%|████████▊ | 8782/10000 [59:42<04:09,  4.88it/s]
 88%|████████▊ | 8783/10000 [59:42<04:10,  4.87it/s]
 88%|████████▊ | 8784/10000 [59:42<04:11,  4.84it/s]
 88%|████████▊ | 8785/10000 [59:43<04:09,  4.88it/s]
 88%|████████▊ | 8786/10000 [59:43<04:02,  5.00it/s]
 88%|████████▊ | 8787/10000 [59:43<03:53,  5.19it/s]
 88%|████████▊ | 8788/10000 [59:43<03:48,  5.31it/s]
 88%|████████▊ | 8789/10000 [59:43<03:43,  5.41it/s]
 88%|████████▊ | 8790/10000 [59:44<03:53,  5.19it/s]
                                                    
{'loss': 0.0545, 'grad_norm': 0.5143530964851379, 'learning_rate': 3.956114316140746e-06}

 88%|████████▊ | 8790/10000 [59:44<03:53,  5.19it/s]
 88%|████████▊ | 8791/10000 [59:44<04:10,  4.83it/s]
 88%|████████▊ | 8792/10000 [59:44<04:16,  4.71it/s]
 88%|████████▊ | 8793/10000 [59:44<04:20,  4.63it/s]
 88%|████████▊ | 8794/10000 [59:44<04:19,  4.65it/s]
 88%|████████▊ | 8795/10000 [59:45<04:05,  4.90it/s]
 88%|████████▊ | 8796/10000 [59:45<03:55,  5.11it/s]
 88%|████████▊ | 8797/10000 [59:45<03:48,  5.26it/s]
 88%|████████▊ | 8798/10000 [59:45<03:46,  5.30it/s]
 88%|████████▊ | 8799/10000 [59:45<03:53,  5.15it/s]
 88%|████████▊ | 8800/10000 [59:46<04:04,  4.90it/s]
                                                    
{'loss': 0.0513, 'grad_norm': 0.6892897486686707, 'learning_rate': 3.891905419629643e-06}

 88%|████████▊ | 8800/10000 [59:46<04:04,  4.90it/s]
 88%|████████▊ | 8801/10000 [59:46<04:16,  4.68it/s]
 88%|████████▊ | 8802/10000 [59:46<05:21,  3.72it/s]
 88%|████████▊ | 8803/10000 [59:46<04:52,  4.09it/s]
 88%|████████▊ | 8804/10000 [59:47<04:29,  4.43it/s]
 88%|████████▊ | 8805/10000 [59:47<04:14,  4.70it/s]
 88%|████████▊ | 8806/10000 [59:47<04:11,  4.75it/s]
 88%|████████▊ | 8807/10000 [59:47<04:15,  4.67it/s]
 88%|████████▊ | 8808/10000 [59:47<04:15,  4.66it/s]
 88%|████████▊ | 8809/10000 [59:48<04:16,  4.64it/s]
 88%|████████▊ | 8810/10000 [59:48<04:16,  4.64it/s]
                                                    
{'loss': 0.0572, 'grad_norm': 0.44503504037857056, 'learning_rate': 3.8282007538536946e-06}

 88%|████████▊ | 8810/10000 [59:48<04:16,  4.64it/s]
 88%|████████▊ | 8811/10000 [59:48<04:05,  4.85it/s]
 88%|████████▊ | 8812/10000 [59:48<03:53,  5.08it/s]
 88%|████████▊ | 8813/10000 [59:48<03:48,  5.20it/s]
 88%|████████▊ | 8814/10000 [59:49<03:52,  5.09it/s]
 88%|████████▊ | 8815/10000 [59:49<03:59,  4.94it/s]
 88%|████████▊ | 8816/10000 [59:49<04:06,  4.81it/s]
 88%|████████▊ | 8817/10000 [59:49<04:11,  4.70it/s]
 88%|████████▊ | 8818/10000 [59:49<04:05,  4.81it/s]
 88%|████████▊ | 8819/10000 [59:50<03:54,  5.03it/s]
 88%|████████▊ | 8820/10000 [59:50<03:46,  5.20it/s]
                                                    
{'loss': 0.0578, 'grad_norm': 0.5227357149124146, 'learning_rate': 3.7650010154769265e-06}

 88%|████████▊ | 8820/10000 [59:50<03:46,  5.20it/s]
 88%|████████▊ | 8821/10000 [59:50<03:45,  5.23it/s]
 88%|████████▊ | 8822/10000 [59:50<03:49,  5.13it/s]
 88%|████████▊ | 8823/10000 [59:50<04:02,  4.85it/s]
 88%|████████▊ | 8824/10000 [59:51<04:04,  4.82it/s]
 88%|████████▊ | 8825/10000 [59:51<04:02,  4.84it/s]
 88%|████████▊ | 8826/10000 [59:51<03:57,  4.94it/s]
 88%|████████▊ | 8827/10000 [59:51<03:47,  5.15it/s]
 88%|████████▊ | 8828/10000 [59:51<03:41,  5.29it/s]
 88%|████████▊ | 8829/10000 [59:52<03:39,  5.34it/s]
 88%|████████▊ | 8830/10000 [59:52<03:45,  5.19it/s]
                                                    
{'loss': 0.0844, 'grad_norm': 0.6890666484832764, 'learning_rate': 3.7023068956415608e-06}

 88%|████████▊ | 8830/10000 [59:52<03:45,  5.19it/s]
 88%|████████▊ | 8831/10000 [59:52<03:59,  4.88it/s]
 88%|████████▊ | 8832/10000 [59:52<04:03,  4.80it/s]
 88%|████████▊ | 8833/10000 [59:52<03:59,  4.87it/s]
 88%|████████▊ | 8834/10000 [59:53<03:51,  5.04it/s]
 88%|████████▊ | 8835/10000 [59:53<03:44,  5.19it/s]
 88%|████████▊ | 8836/10000 [59:53<03:39,  5.30it/s]
 88%|████████▊ | 8837/10000 [59:53<03:41,  5.25it/s]
 88%|████████▊ | 8838/10000 [59:53<03:45,  5.16it/s]
 88%|████████▊ | 8839/10000 [59:54<03:51,  5.02it/s]
 88%|████████▊ | 8840/10000 [59:54<04:03,  4.76it/s]
                                                    
{'loss': 0.065, 'grad_norm': 0.5341351628303528, 'learning_rate': 3.6401190799604303e-06}

 88%|████████▊ | 8840/10000 [59:54<04:03,  4.76it/s]
 88%|████████▊ | 8841/10000 [59:54<04:15,  4.54it/s]
 88%|████████▊ | 8842/10000 [59:54<04:11,  4.60it/s]
 88%|████████▊ | 8843/10000 [59:54<04:01,  4.79it/s]
 88%|████████▊ | 8844/10000 [59:55<03:51,  5.00it/s]
 88%|████████▊ | 8845/10000 [59:55<03:48,  5.06it/s]
 88%|████████▊ | 8846/10000 [59:55<03:49,  5.03it/s]
 88%|████████▊ | 8847/10000 [59:55<03:53,  4.95it/s]
 88%|████████▊ | 8848/10000 [59:55<04:00,  4.78it/s]
 88%|████████▊ | 8849/10000 [59:56<03:59,  4.81it/s]
 88%|████████▊ | 8850/10000 [59:56<04:04,  4.70it/s]
                                                    
{'loss': 0.1083, 'grad_norm': 0.7025235295295715, 'learning_rate': 3.578438248509536e-06}

 88%|████████▊ | 8850/10000 [59:56<04:04,  4.70it/s]
 89%|████████▊ | 8851/10000 [59:56<04:08,  4.62it/s]
 89%|████████▊ | 8852/10000 [59:56<04:06,  4.65it/s]
 89%|████████▊ | 8853/10000 [59:56<03:58,  4.81it/s]
 89%|████████▊ | 8854/10000 [59:57<03:51,  4.94it/s]
 89%|████████▊ | 8855/10000 [59:57<03:45,  5.08it/s]
 89%|████████▊ | 8856/10000 [59:57<03:48,  5.01it/s]
 89%|████████▊ | 8857/10000 [59:57<03:57,  4.81it/s]
 89%|████████▊ | 8858/10000 [59:58<04:02,  4.70it/s]
 89%|████████▊ | 8859/10000 [59:58<03:59,  4.76it/s]
 89%|████████▊ | 8860/10000 [59:58<04:02,  4.71it/s]
                                                    
{'loss': 0.0893, 'grad_norm': 0.6796401143074036, 'learning_rate': 3.5172650758205583e-06}

 89%|████████▊ | 8860/10000 [59:58<04:02,  4.71it/s]
 89%|████████▊ | 8861/10000 [59:58<03:58,  4.77it/s]
 89%|████████▊ | 8862/10000 [59:58<03:48,  4.97it/s]
 89%|████████▊ | 8863/10000 [59:59<03:43,  5.09it/s]
 89%|████████▊ | 8864/10000 [59:59<03:38,  5.21it/s]
 89%|████████▊ | 8865/10000 [59:59<03:39,  5.18it/s]
 89%|████████▊ | 8866/10000 [59:59<03:41,  5.13it/s]
 89%|████████▊ | 8867/10000 [59:59<03:44,  5.04it/s]
 89%|████████▊ | 8868/10000 [1:00:00<03:46,  5.00it/s]
 89%|████████▊ | 8869/10000 [1:00:00<03:45,  5.01it/s]
 89%|████████▊ | 8870/10000 [1:00:00<03:45,  5.02it/s]
                                                      
{'loss': 0.0792, 'grad_norm': 0.580812394618988, 'learning_rate': 3.45660023087353e-06}

 89%|████████▊ | 8870/10000 [1:00:00<03:45,  5.02it/s]
 89%|████████▊ | 8871/10000 [1:00:00<03:44,  5.03it/s]
 89%|████████▊ | 8872/10000 [1:00:00<03:38,  5.16it/s]
 89%|████████▊ | 8873/10000 [1:00:00<03:36,  5.21it/s]
 89%|████████▊ | 8874/10000 [1:00:01<03:33,  5.26it/s]
 89%|████████▉ | 8875/10000 [1:00:01<03:40,  5.11it/s]
 89%|████████▉ | 8876/10000 [1:00:01<03:47,  4.95it/s]
 89%|████████▉ | 8877/10000 [1:00:01<03:50,  4.87it/s]
 89%|████████▉ | 8878/10000 [1:00:02<03:56,  4.74it/s]
 89%|████████▉ | 8879/10000 [1:00:02<03:51,  4.85it/s]
 89%|████████▉ | 8880/10000 [1:00:02<03:44,  5.00it/s]
                                                      
{'loss': 0.0656, 'grad_norm': 0.5423473119735718, 'learning_rate': 3.3964443770894528e-06}

 89%|████████▉ | 8880/10000 [1:00:02<03:44,  5.00it/s]
 89%|████████▉ | 8881/10000 [1:00:02<03:40,  5.08it/s]
 89%|████████▉ | 8882/10000 [1:00:02<03:36,  5.16it/s]
 89%|████████▉ | 8883/10000 [1:00:02<03:33,  5.23it/s]
 89%|████████▉ | 8884/10000 [1:00:03<03:31,  5.27it/s]
 89%|████████▉ | 8885/10000 [1:00:03<03:33,  5.23it/s]
 89%|████████▉ | 8886/10000 [1:00:03<03:37,  5.11it/s]
 89%|████████▉ | 8887/10000 [1:00:03<03:42,  4.99it/s]
 89%|████████▉ | 8888/10000 [1:00:03<03:43,  4.97it/s]
 89%|████████▉ | 8889/10000 [1:00:04<03:47,  4.89it/s]
 89%|████████▉ | 8890/10000 [1:00:04<03:45,  4.92it/s]
                                                      
{'loss': 0.0909, 'grad_norm': 0.8653892874717712, 'learning_rate': 3.3367981723231245e-06}

 89%|████████▉ | 8890/10000 [1:00:04<03:45,  4.92it/s]
 89%|████████▉ | 8891/10000 [1:00:04<03:46,  4.89it/s]
 89%|████████▉ | 8892/10000 [1:00:04<03:44,  4.94it/s]
 89%|████████▉ | 8893/10000 [1:00:04<03:45,  4.92it/s]
 89%|████████▉ | 8894/10000 [1:00:05<03:48,  4.84it/s]
 89%|████████▉ | 8895/10000 [1:00:05<03:47,  4.85it/s]
 89%|████████▉ | 8896/10000 [1:00:05<03:53,  4.73it/s]
 89%|████████▉ | 8897/10000 [1:00:05<04:04,  4.51it/s]
 89%|████████▉ | 8898/10000 [1:00:06<04:05,  4.48it/s]
 89%|████████▉ | 8899/10000 [1:00:06<04:07,  4.45it/s]
 89%|████████▉ | 8900/10000 [1:00:06<04:16,  4.30it/s]
                                                      
{'loss': 0.0688, 'grad_norm': 0.6121671795845032, 'learning_rate': 3.2776622688558746e-06}

 89%|████████▉ | 8900/10000 [1:00:06<04:16,  4.30it/s]
 89%|████████▉ | 8901/10000 [1:00:06<04:24,  4.16it/s]
 89%|████████▉ | 8902/10000 [1:00:07<04:09,  4.39it/s]
 89%|████████▉ | 8903/10000 [1:00:07<04:01,  4.55it/s]
 89%|████████▉ | 8904/10000 [1:00:07<04:02,  4.53it/s]
 89%|████████▉ | 8905/10000 [1:00:07<04:03,  4.50it/s]
 89%|████████▉ | 8906/10000 [1:00:07<03:57,  4.60it/s]
 89%|████████▉ | 8907/10000 [1:00:08<04:04,  4.48it/s]
 89%|████████▉ | 8908/10000 [1:00:08<04:12,  4.32it/s]
 89%|████████▉ | 8909/10000 [1:00:08<04:04,  4.46it/s]
 89%|████████▉ | 8910/10000 [1:00:08<04:00,  4.54it/s]
                                                      
{'loss': 0.0706, 'grad_norm': 0.5546550750732422, 'learning_rate': 3.2190373133884677e-06}

 89%|████████▉ | 8910/10000 [1:00:08<04:00,  4.54it/s]
 89%|████████▉ | 8911/10000 [1:00:09<04:00,  4.52it/s]
 89%|████████▉ | 8912/10000 [1:00:09<03:57,  4.58it/s]
 89%|████████▉ | 8913/10000 [1:00:09<03:56,  4.60it/s]
 89%|████████▉ | 8914/10000 [1:00:09<03:53,  4.65it/s]
 89%|████████▉ | 8915/10000 [1:00:09<03:46,  4.79it/s]
 89%|████████▉ | 8916/10000 [1:00:10<03:40,  4.93it/s]
 89%|████████▉ | 8917/10000 [1:00:10<03:50,  4.71it/s]
 89%|████████▉ | 8918/10000 [1:00:10<03:55,  4.60it/s]
 89%|████████▉ | 8919/10000 [1:00:10<03:56,  4.58it/s]
 89%|████████▉ | 8920/10000 [1:00:10<03:56,  4.56it/s]
                                                      
{'loss': 0.0751, 'grad_norm': 0.6025184392929077, 'learning_rate': 3.1609239470340446e-06}

 89%|████████▉ | 8920/10000 [1:00:10<03:56,  4.56it/s]
 89%|████████▉ | 8921/10000 [1:00:11<03:57,  4.55it/s]
 89%|████████▉ | 8922/10000 [1:00:11<03:52,  4.64it/s]
 89%|████████▉ | 8923/10000 [1:00:11<03:43,  4.81it/s]
 89%|████████▉ | 8924/10000 [1:00:11<03:36,  4.98it/s]
 89%|████████▉ | 8925/10000 [1:00:11<03:40,  4.88it/s]
 89%|████████▉ | 8926/10000 [1:00:12<03:43,  4.80it/s]
 89%|████████▉ | 8927/10000 [1:00:12<03:47,  4.73it/s]
 89%|████████▉ | 8928/10000 [1:00:12<03:55,  4.56it/s]
 89%|████████▉ | 8929/10000 [1:00:12<03:55,  4.54it/s]
 89%|████████▉ | 8930/10000 [1:00:13<03:54,  4.57it/s]
                                                      
{'loss': 0.0613, 'grad_norm': 0.8763092160224915, 'learning_rate': 3.1033228053110373e-06}

 89%|████████▉ | 8930/10000 [1:00:13<03:54,  4.57it/s]
 89%|████████▉ | 8931/10000 [1:00:13<03:49,  4.66it/s]
 89%|████████▉ | 8932/10000 [1:00:13<03:45,  4.74it/s]
 89%|████████▉ | 8933/10000 [1:00:13<03:43,  4.77it/s]
 89%|████████▉ | 8934/10000 [1:00:13<03:46,  4.71it/s]
 89%|████████▉ | 8935/10000 [1:00:14<03:53,  4.57it/s]
 89%|████████▉ | 8936/10000 [1:00:14<03:53,  4.55it/s]
 89%|████████▉ | 8937/10000 [1:00:14<03:54,  4.53it/s]
 89%|████████▉ | 8938/10000 [1:00:14<03:53,  4.55it/s]
 89%|████████▉ | 8939/10000 [1:00:15<03:52,  4.57it/s]
 89%|████████▉ | 8940/10000 [1:00:15<03:52,  4.56it/s]
                                                      
{'loss': 0.0509, 'grad_norm': 0.5198410749435425, 'learning_rate': 3.0462345181363314e-06}

 89%|████████▉ | 8940/10000 [1:00:15<03:52,  4.56it/s]
 89%|████████▉ | 8941/10000 [1:00:15<03:57,  4.46it/s]
 89%|████████▉ | 8942/10000 [1:00:15<03:59,  4.42it/s]
 89%|████████▉ | 8943/10000 [1:00:15<04:09,  4.24it/s]
 89%|████████▉ | 8944/10000 [1:00:16<04:08,  4.24it/s]
 89%|████████▉ | 8945/10000 [1:00:16<04:11,  4.20it/s]
 89%|████████▉ | 8946/10000 [1:00:16<04:00,  4.39it/s]
 89%|████████▉ | 8947/10000 [1:00:16<03:47,  4.64it/s]
 89%|████████▉ | 8948/10000 [1:00:17<03:40,  4.76it/s]
 89%|████████▉ | 8949/10000 [1:00:17<03:39,  4.79it/s]
 90%|████████▉ | 8950/10000 [1:00:17<03:39,  4.79it/s]
                                                      
{'loss': 0.0952, 'grad_norm': 0.6058054566383362, 'learning_rate': 2.9896597098182654e-06}

 90%|████████▉ | 8950/10000 [1:00:17<03:39,  4.79it/s]
 90%|████████▉ | 8951/10000 [1:00:17<03:49,  4.56it/s]
 90%|████████▉ | 8952/10000 [1:00:17<03:56,  4.43it/s]
 90%|████████▉ | 8953/10000 [1:00:18<04:04,  4.28it/s]
 90%|████████▉ | 8954/10000 [1:00:18<04:02,  4.32it/s]
 90%|████████▉ | 8955/10000 [1:00:18<03:49,  4.56it/s]
 90%|████████▉ | 8956/10000 [1:00:18<03:40,  4.73it/s]
 90%|████████▉ | 8957/10000 [1:00:18<03:32,  4.92it/s]
 90%|████████▉ | 8958/10000 [1:00:19<03:34,  4.87it/s]
 90%|████████▉ | 8959/10000 [1:00:19<03:45,  4.62it/s]
 90%|████████▉ | 8960/10000 [1:00:19<03:55,  4.42it/s]
                                                      
{'loss': 0.0672, 'grad_norm': 0.7207470536231995, 'learning_rate': 2.933598999049891e-06}

 90%|████████▉ | 8960/10000 [1:00:19<03:55,  4.42it/s]
 90%|████████▉ | 8961/10000 [1:00:19<04:07,  4.20it/s]
 90%|████████▉ | 8962/10000 [1:00:20<04:07,  4.20it/s]
 90%|████████▉ | 8963/10000 [1:00:20<03:55,  4.40it/s]
 90%|████████▉ | 8964/10000 [1:00:20<03:47,  4.56it/s]
 90%|████████▉ | 8965/10000 [1:00:20<03:38,  4.74it/s]
 90%|████████▉ | 8966/10000 [1:00:20<03:34,  4.82it/s]
 90%|████████▉ | 8967/10000 [1:00:21<03:34,  4.81it/s]
 90%|████████▉ | 8968/10000 [1:00:21<03:48,  4.51it/s]
 90%|████████▉ | 8969/10000 [1:00:21<03:46,  4.55it/s]
 90%|████████▉ | 8970/10000 [1:00:21<03:47,  4.53it/s]
                                                      
{'loss': 0.0742, 'grad_norm': 0.5980435013771057, 'learning_rate': 2.8780529989021697e-06}

 90%|████████▉ | 8970/10000 [1:00:21<03:47,  4.53it/s]
 90%|████████▉ | 8971/10000 [1:00:22<03:46,  4.53it/s]
 90%|████████▉ | 8972/10000 [1:00:22<03:39,  4.67it/s]
 90%|████████▉ | 8973/10000 [1:00:22<03:29,  4.90it/s]
 90%|████████▉ | 8974/10000 [1:00:22<03:25,  4.99it/s]
 90%|████████▉ | 8975/10000 [1:00:22<03:25,  4.98it/s]
 90%|████████▉ | 8976/10000 [1:00:23<03:28,  4.92it/s]
 90%|████████▉ | 8977/10000 [1:00:23<03:43,  4.57it/s]
 90%|████████▉ | 8978/10000 [1:00:23<03:56,  4.31it/s]
 90%|████████▉ | 8979/10000 [1:00:23<04:01,  4.23it/s]
 90%|████████▉ | 8980/10000 [1:00:24<03:58,  4.28it/s]
                                                      
{'loss': 0.0493, 'grad_norm': 0.5771141648292542, 'learning_rate': 2.823022316817242e-06}

 90%|████████▉ | 8980/10000 [1:00:24<03:58,  4.28it/s]
 90%|████████▉ | 8981/10000 [1:00:24<03:50,  4.42it/s]
 90%|████████▉ | 8982/10000 [1:00:24<03:41,  4.60it/s]
 90%|████████▉ | 8983/10000 [1:00:24<03:34,  4.75it/s]
 90%|████████▉ | 8984/10000 [1:00:24<03:33,  4.77it/s]
 90%|████████▉ | 8985/10000 [1:00:25<03:31,  4.79it/s]
 90%|████████▉ | 8986/10000 [1:00:25<03:35,  4.71it/s]
 90%|████████▉ | 8987/10000 [1:00:25<03:42,  4.55it/s]
 90%|████████▉ | 8988/10000 [1:00:25<03:43,  4.53it/s]
 90%|████████▉ | 8989/10000 [1:00:25<03:41,  4.57it/s]
 90%|████████▉ | 8990/10000 [1:00:26<03:38,  4.61it/s]
                                                      
{'loss': 0.0685, 'grad_norm': 0.6250346899032593, 'learning_rate': 2.7685075546018456e-06}

 90%|████████▉ | 8990/10000 [1:00:26<03:38,  4.61it/s]
 90%|████████▉ | 8991/10000 [1:00:26<03:40,  4.57it/s]
 90%|████████▉ | 8992/10000 [1:00:26<03:35,  4.68it/s]
 90%|████████▉ | 8993/10000 [1:00:26<03:32,  4.74it/s]
 90%|████████▉ | 8994/10000 [1:00:27<03:30,  4.77it/s]
 90%|████████▉ | 8995/10000 [1:00:27<03:29,  4.80it/s]
 90%|████████▉ | 8996/10000 [1:00:27<03:29,  4.80it/s]
 90%|████████▉ | 8997/10000 [1:00:27<03:33,  4.70it/s]
 90%|████████▉ | 8998/10000 [1:00:27<03:37,  4.61it/s]
 90%|████████▉ | 8999/10000 [1:00:28<03:40,  4.55it/s]
 90%|█████████ | 9000/10000 [1:00:28<03:40,  4.54it/s]Rank 0, Worker 0: Caching shard...Rank 0, Worker 1: Caching shard...

Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Caching shard...Rank 0, Worker 4: Caching shard...

Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.81 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.17 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.23 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.45 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.71 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.79 seconds
Rank 0, Worker 2: Caching shard...
[held-out-eval] step 9000: eval/loss = 0.0856 (35s)

                                                      
{'loss': 0.085, 'grad_norm': 0.5154876708984375, 'learning_rate': 2.7145093084206598e-06}

 90%|█████████ | 9000/10000 [1:01:03<03:40,  4.54it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-9000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-9000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-9000/wandb_config.json

 90%|█████████ | 9001/10000 [1:01:32<5:20:51, 19.27s/it]
 90%|█████████ | 9002/10000 [1:01:32<3:45:30, 13.56s/it]
 90%|█████████ | 9003/10000 [1:01:32<2:38:41,  9.55s/it]
 90%|█████████ | 9004/10000 [1:01:32<1:51:54,  6.74s/it]
 90%|█████████ | 9005/10000 [1:01:32<1:19:13,  4.78s/it]
 90%|█████████ | 9006/10000 [1:01:33<56:27,  3.41s/it]  
 90%|█████████ | 9007/10000 [1:01:33<40:29,  2.45s/it]
 90%|█████████ | 9008/10000 [1:01:33<29:27,  1.78s/it]
 90%|█████████ | 9009/10000 [1:01:33<21:46,  1.32s/it]
 90%|█████████ | 9010/10000 [1:01:33<16:19,  1.01it/s]
                                                      
{'loss': 0.074, 'grad_norm': 0.6177858114242554, 'learning_rate': 2.661028168789892e-06}

 90%|█████████ | 9010/10000 [1:01:34<16:19,  1.01it/s]
 90%|█████████ | 9011/10000 [1:01:34<12:34,  1.31it/s]
 90%|█████████ | 9012/10000 [1:01:34<09:53,  1.67it/s]
 90%|█████████ | 9013/10000 [1:01:34<07:52,  2.09it/s]Rank 0, Worker 1: Wait for shard 44 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 90%|█████████ | 9014/10000 [1:01:34<06:31,  2.52it/s]
 90%|█████████ | 9015/10000 [1:01:35<05:32,  2.96it/s]
 90%|█████████ | 9016/10000 [1:01:35<04:52,  3.37it/s]
 90%|█████████ | 9017/10000 [1:01:35<04:24,  3.72it/s]
 90%|█████████ | 9018/10000 [1:01:35<04:08,  3.95it/s]
 90%|█████████ | 9019/10000 [1:01:35<03:59,  4.10it/s]
 90%|█████████ | 9020/10000 [1:01:36<03:55,  4.16it/s]
                                                      
{'loss': 0.0555, 'grad_norm': 0.5644664168357849, 'learning_rate': 2.6080647205706855e-06}

 90%|█████████ | 9020/10000 [1:01:36<03:55,  4.16it/s]
 90%|█████████ | 9021/10000 [1:01:36<03:52,  4.21it/s]
 90%|█████████ | 9022/10000 [1:01:36<03:46,  4.32it/s]
 90%|█████████ | 9023/10000 [1:01:36<03:36,  4.52it/s]
 90%|█████████ | 9024/10000 [1:01:36<03:24,  4.77it/s]
 90%|█████████ | 9025/10000 [1:01:37<03:17,  4.95it/s]
 90%|█████████ | 9026/10000 [1:01:37<03:11,  5.08it/s]
 90%|█████████ | 9027/10000 [1:01:37<03:15,  4.97it/s]
 90%|█████████ | 9028/10000 [1:01:37<03:22,  4.81it/s]
 90%|█████████ | 9029/10000 [1:01:37<03:36,  4.49it/s]
 90%|█████████ | 9030/10000 [1:01:38<03:39,  4.41it/s]
                                                      
{'loss': 0.0674, 'grad_norm': 0.4792957901954651, 'learning_rate': 2.555619542962834e-06}

 90%|█████████ | 9030/10000 [1:01:38<03:39,  4.41it/s]
 90%|█████████ | 9031/10000 [1:01:38<03:42,  4.35it/s]
 90%|█████████ | 9032/10000 [1:01:38<03:35,  4.48it/s]
 90%|█████████ | 9033/10000 [1:01:38<03:27,  4.66it/s]
 90%|█████████ | 9034/10000 [1:01:39<03:32,  4.54it/s]
 90%|█████████ | 9035/10000 [1:01:39<03:26,  4.66it/s]
 90%|█████████ | 9036/10000 [1:01:39<03:35,  4.47it/s]
 90%|█████████ | 9037/10000 [1:01:39<03:42,  4.33it/s]
 90%|█████████ | 9038/10000 [1:01:40<03:52,  4.14it/s]
 90%|█████████ | 9039/10000 [1:01:40<03:53,  4.12it/s]
 90%|█████████ | 9040/10000 [1:01:40<03:48,  4.19it/s]
                                                      
{'loss': 0.0739, 'grad_norm': 0.8346887230873108, 'learning_rate': 2.503693209498409e-06}

 90%|█████████ | 9040/10000 [1:01:40<03:48,  4.19it/s]
 90%|█████████ | 9041/10000 [1:01:40<03:37,  4.41it/s]
 90%|█████████ | 9042/10000 [1:01:40<03:24,  4.69it/s]
 90%|█████████ | 9043/10000 [1:01:41<03:22,  4.72it/s]
 90%|█████████ | 9044/10000 [1:01:41<03:21,  4.74it/s]
 90%|█████████ | 9045/10000 [1:01:41<03:23,  4.69it/s]
 90%|█████████ | 9046/10000 [1:01:41<03:29,  4.55it/s]
 90%|█████████ | 9047/10000 [1:01:42<03:33,  4.47it/s]
 90%|█████████ | 9048/10000 [1:01:42<03:27,  4.58it/s]
 90%|█████████ | 9049/10000 [1:01:42<03:23,  4.68it/s]
 90%|█████████ | 9050/10000 [1:01:42<03:22,  4.69it/s]
                                                      
{'loss': 0.07, 'grad_norm': 0.8148089647293091, 'learning_rate': 2.452286288035449e-06}

 90%|█████████ | 9050/10000 [1:01:42<03:22,  4.69it/s]
 91%|█████████ | 9051/10000 [1:01:42<03:23,  4.66it/s]
 91%|█████████ | 9052/10000 [1:01:43<03:21,  4.71it/s]
 91%|█████████ | 9053/10000 [1:01:43<03:23,  4.65it/s]
 91%|█████████ | 9054/10000 [1:01:43<03:26,  4.58it/s]
 91%|█████████ | 9055/10000 [1:01:43<03:44,  4.20it/s]
 91%|█████████ | 9056/10000 [1:01:44<03:48,  4.13it/s]
 91%|█████████ | 9057/10000 [1:01:44<03:45,  4.18it/s]
 91%|█████████ | 9058/10000 [1:01:44<03:35,  4.37it/s]
 91%|█████████ | 9059/10000 [1:01:44<03:35,  4.37it/s]
 91%|█████████ | 9060/10000 [1:01:44<03:39,  4.28it/s]
                                                      
{'loss': 0.0583, 'grad_norm': 0.5610070824623108, 'learning_rate': 2.4013993407518363e-06}

 91%|█████████ | 9060/10000 [1:01:45<03:39,  4.28it/s]
 91%|█████████ | 9061/10000 [1:01:45<03:40,  4.26it/s]
 91%|█████████ | 9062/10000 [1:01:45<03:39,  4.28it/s]
 91%|█████████ | 9063/10000 [1:01:45<03:37,  4.30it/s]
 91%|█████████ | 9064/10000 [1:01:45<03:33,  4.39it/s]
 91%|█████████ | 9065/10000 [1:01:46<03:30,  4.44it/s]
 91%|█████████ | 9066/10000 [1:01:46<03:27,  4.51it/s]
 91%|█████████ | 9067/10000 [1:01:46<03:30,  4.43it/s]
 91%|█████████ | 9068/10000 [1:01:46<03:28,  4.48it/s]
 91%|█████████ | 9069/10000 [1:01:46<03:28,  4.46it/s]
 91%|█████████ | 9070/10000 [1:01:47<03:24,  4.55it/s]
                                                      
{'loss': 0.0809, 'grad_norm': 0.4932200312614441, 'learning_rate': 2.351032924139063e-06}

 91%|█████████ | 9070/10000 [1:01:47<03:24,  4.55it/s]
 91%|█████████ | 9071/10000 [1:01:47<03:28,  4.47it/s]
 91%|█████████ | 9072/10000 [1:01:47<03:25,  4.51it/s]
 91%|█████████ | 9073/10000 [1:01:47<03:24,  4.54it/s]
 91%|█████████ | 9074/10000 [1:01:48<03:23,  4.56it/s]
 91%|█████████ | 9075/10000 [1:01:48<03:25,  4.50it/s]
 91%|█████████ | 9076/10000 [1:01:48<03:25,  4.50it/s]
 91%|█████████ | 9077/10000 [1:01:48<03:27,  4.46it/s]
 91%|█████████ | 9078/10000 [1:01:49<03:32,  4.35it/s]
 91%|█████████ | 9079/10000 [1:01:49<03:38,  4.22it/s]
 91%|█████████ | 9080/10000 [1:01:49<03:38,  4.20it/s]
                                                      
{'loss': 0.0559, 'grad_norm': 0.590142011642456, 'learning_rate': 2.30118758899619e-06}

 91%|█████████ | 9080/10000 [1:01:49<03:38,  4.20it/s]
 91%|█████████ | 9081/10000 [1:01:49<03:42,  4.14it/s]
 91%|█████████ | 9082/10000 [1:01:49<03:38,  4.20it/s]
 91%|█████████ | 9083/10000 [1:01:50<03:41,  4.14it/s]
 91%|█████████ | 9084/10000 [1:01:50<03:29,  4.37it/s]
 91%|█████████ | 9085/10000 [1:01:50<03:26,  4.42it/s]
 91%|█████████ | 9086/10000 [1:01:50<03:31,  4.32it/s]Rank 0, Worker 2: Wait for shard 13 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 91%|█████████ | 9087/10000 [1:01:51<03:35,  4.23it/s]
 91%|█████████ | 9088/10000 [1:01:51<03:43,  4.09it/s]
 91%|█████████ | 9089/10000 [1:01:51<03:46,  4.02it/s]
 91%|█████████ | 9090/10000 [1:01:51<03:49,  3.97it/s]
                                                      
{'loss': 0.0855, 'grad_norm': 0.6319729685783386, 'learning_rate': 2.2518638804238157e-06}

 91%|█████████ | 9090/10000 [1:01:51<03:49,  3.97it/s]
 91%|█████████ | 9091/10000 [1:01:52<03:46,  4.01it/s]
 91%|█████████ | 9092/10000 [1:01:52<03:43,  4.06it/s]
 91%|█████████ | 9093/10000 [1:01:52<03:37,  4.16it/s]
 91%|█████████ | 9094/10000 [1:01:52<03:32,  4.26it/s]
 91%|█████████ | 9095/10000 [1:01:53<03:28,  4.35it/s]
 91%|█████████ | 9096/10000 [1:01:53<03:33,  4.23it/s]
 91%|█████████ | 9097/10000 [1:01:53<03:34,  4.22it/s]
 91%|█████████ | 9098/10000 [1:01:53<03:30,  4.29it/s]
 91%|█████████ | 9099/10000 [1:01:54<03:37,  4.15it/s]
 91%|█████████ | 9100/10000 [1:01:54<03:35,  4.18it/s]
                                                      
{'loss': 0.0653, 'grad_norm': 0.7253748178482056, 'learning_rate': 2.203062337818118e-06}

 91%|█████████ | 9100/10000 [1:01:54<03:35,  4.18it/s]
 91%|█████████ | 9101/10000 [1:01:54<03:28,  4.31it/s]
 91%|█████████ | 9102/10000 [1:01:54<03:23,  4.41it/s]
 91%|█████████ | 9103/10000 [1:01:54<03:11,  4.68it/s]
 91%|█████████ | 9104/10000 [1:01:55<03:05,  4.82it/s]
 91%|█████████ | 9105/10000 [1:01:55<03:07,  4.77it/s]
 91%|█████████ | 9106/10000 [1:01:55<03:09,  4.72it/s]
 91%|█████████ | 9107/10000 [1:01:55<03:14,  4.60it/s]
 91%|█████████ | 9108/10000 [1:01:55<03:21,  4.44it/s]
 91%|█████████ | 9109/10000 [1:01:56<03:25,  4.34it/s]
 91%|█████████ | 9110/10000 [1:01:56<03:20,  4.44it/s]
                                                      
{'loss': 0.0699, 'grad_norm': 0.5912812948226929, 'learning_rate': 2.1547834948649483e-06}

 91%|█████████ | 9110/10000 [1:01:56<03:20,  4.44it/s]
 91%|█████████ | 9111/10000 [1:01:56<03:15,  4.55it/s]
 91%|█████████ | 9112/10000 [1:01:56<03:05,  4.78it/s]
 91%|█████████ | 9113/10000 [1:01:57<03:04,  4.82it/s]
 91%|█████████ | 9114/10000 [1:01:57<03:03,  4.82it/s]
 91%|█████████ | 9115/10000 [1:01:57<03:11,  4.62it/s]
 91%|█████████ | 9116/10000 [1:01:57<03:21,  4.40it/s]
 91%|█████████ | 9117/10000 [1:01:57<03:23,  4.33it/s]
 91%|█████████ | 9118/10000 [1:01:58<03:16,  4.49it/s]
 91%|█████████ | 9119/10000 [1:01:58<03:09,  4.64it/s]
 91%|█████████ | 9120/10000 [1:01:58<03:07,  4.69it/s]
                                                      
{'loss': 0.0783, 'grad_norm': 0.5556371808052063, 'learning_rate': 2.1070278795340017e-06}

 91%|█████████ | 9120/10000 [1:01:58<03:07,  4.69it/s]
 91%|█████████ | 9121/10000 [1:01:58<03:09,  4.64it/s]
 91%|█████████ | 9122/10000 [1:01:59<03:10,  4.61it/s]
 91%|█████████ | 9123/10000 [1:01:59<03:11,  4.59it/s]
 91%|█████████ | 9124/10000 [1:01:59<03:13,  4.54it/s]Rank 0, Worker 4: Wait for shard 15 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 91%|█████████▏| 9125/10000 [1:01:59<03:11,  4.58it/s]
 91%|█████████▏| 9126/10000 [1:01:59<03:16,  4.45it/s]
 91%|█████████▏| 9127/10000 [1:02:00<03:14,  4.49it/s]
 91%|█████████▏| 9128/10000 [1:02:00<03:06,  4.67it/s]
 91%|█████████▏| 9129/10000 [1:02:00<03:03,  4.75it/s]
 91%|█████████▏| 9130/10000 [1:02:00<03:01,  4.80it/s]
                                                      
{'loss': 0.0639, 'grad_norm': 0.7622410655021667, 'learning_rate': 2.059796014073029e-06}

 91%|█████████▏| 9130/10000 [1:02:00<03:01,  4.80it/s]
 91%|█████████▏| 9131/10000 [1:02:00<03:10,  4.56it/s]
 91%|█████████▏| 9132/10000 [1:02:01<03:16,  4.42it/s]
 91%|█████████▏| 9133/10000 [1:02:01<03:10,  4.55it/s]
 91%|█████████▏| 9134/10000 [1:02:01<03:08,  4.60it/s]
 91%|█████████▏| 9135/10000 [1:02:01<03:09,  4.55it/s]
 91%|█████████▏| 9136/10000 [1:02:02<03:00,  4.79it/s]
 91%|█████████▏| 9137/10000 [1:02:02<02:55,  4.91it/s]
 91%|█████████▏| 9138/10000 [1:02:02<02:56,  4.89it/s]
 91%|█████████▏| 9139/10000 [1:02:02<02:56,  4.87it/s]
 91%|█████████▏| 9140/10000 [1:02:02<02:56,  4.87it/s]
                                                      
{'loss': 0.0857, 'grad_norm': 0.6412494778633118, 'learning_rate': 2.01308841500214e-06}

 91%|█████████▏| 9140/10000 [1:02:02<02:56,  4.87it/s]
 91%|█████████▏| 9141/10000 [1:02:03<02:58,  4.80it/s]
 91%|█████████▏| 9142/10000 [1:02:03<02:57,  4.82it/s]
 91%|█████████▏| 9143/10000 [1:02:03<02:56,  4.86it/s]
 91%|█████████▏| 9144/10000 [1:02:03<02:52,  4.95it/s]
 91%|█████████▏| 9145/10000 [1:02:03<02:50,  5.01it/s]
 91%|█████████▏| 9146/10000 [1:02:04<02:51,  4.99it/s]
 91%|█████████▏| 9147/10000 [1:02:04<02:53,  4.92it/s]
 91%|█████████▏| 9148/10000 [1:02:04<02:59,  4.74it/s]
 91%|█████████▏| 9149/10000 [1:02:04<02:57,  4.79it/s]
 92%|█████████▏| 9150/10000 [1:02:04<02:57,  4.80it/s]
                                                      
{'loss': 0.0633, 'grad_norm': 0.7338694334030151, 'learning_rate': 1.9669055931081704e-06}

 92%|█████████▏| 9150/10000 [1:02:04<02:57,  4.80it/s]
 92%|█████████▏| 9151/10000 [1:02:05<03:02,  4.65it/s]
 92%|█████████▏| 9152/10000 [1:02:05<02:59,  4.73it/s]
 92%|█████████▏| 9153/10000 [1:02:05<02:56,  4.79it/s]
 92%|█████████▏| 9154/10000 [1:02:05<02:55,  4.82it/s]
 92%|█████████▏| 9155/10000 [1:02:05<02:51,  4.91it/s]
 92%|█████████▏| 9156/10000 [1:02:06<02:50,  4.94it/s]
 92%|█████████▏| 9157/10000 [1:02:06<02:50,  4.95it/s]
 92%|█████████▏| 9158/10000 [1:02:06<02:46,  5.04it/s]
 92%|█████████▏| 9159/10000 [1:02:06<02:44,  5.12it/s]
 92%|█████████▏| 9160/10000 [1:02:06<02:45,  5.07it/s]
                                                      
{'loss': 0.063, 'grad_norm': 0.7937187552452087, 'learning_rate': 1.9212480534390507e-06}

 92%|█████████▏| 9160/10000 [1:02:06<02:45,  5.07it/s]
 92%|█████████▏| 9161/10000 [1:02:07<02:47,  4.99it/s]
 92%|█████████▏| 9162/10000 [1:02:07<02:51,  4.90it/s]
 92%|█████████▏| 9163/10000 [1:02:07<02:51,  4.88it/s]
 92%|█████████▏| 9164/10000 [1:02:07<02:50,  4.91it/s]
 92%|█████████▏| 9165/10000 [1:02:07<02:49,  4.91it/s]
 92%|█████████▏| 9166/10000 [1:02:08<02:47,  4.97it/s]
 92%|█████████▏| 9167/10000 [1:02:08<02:45,  5.02it/s]
 92%|█████████▏| 9168/10000 [1:02:08<02:44,  5.06it/s]
 92%|█████████▏| 9169/10000 [1:02:08<02:42,  5.10it/s]
 92%|█████████▏| 9170/10000 [1:02:08<02:43,  5.07it/s]
                                                      
{'loss': 0.0646, 'grad_norm': 0.5113723874092102, 'learning_rate': 1.8761162952983246e-06}

 92%|█████████▏| 9170/10000 [1:02:09<02:43,  5.07it/s]
 92%|█████████▏| 9171/10000 [1:02:09<02:46,  4.98it/s]
 92%|█████████▏| 9172/10000 [1:02:09<02:45,  4.99it/s]
 92%|█████████▏| 9173/10000 [1:02:09<02:45,  5.00it/s]
 92%|█████████▏| 9174/10000 [1:02:09<02:46,  4.97it/s]
 92%|█████████▏| 9175/10000 [1:02:09<02:46,  4.96it/s]
 92%|█████████▏| 9176/10000 [1:02:10<02:45,  4.97it/s]
 92%|█████████▏| 9177/10000 [1:02:10<02:47,  4.90it/s]
 92%|█████████▏| 9178/10000 [1:02:10<02:46,  4.94it/s]
 92%|█████████▏| 9179/10000 [1:02:10<02:46,  4.93it/s]
 92%|█████████▏| 9180/10000 [1:02:10<02:43,  5.01it/s]
                                                      
{'loss': 0.0559, 'grad_norm': 0.5560367703437805, 'learning_rate': 1.8315108122396618e-06}

 92%|█████████▏| 9180/10000 [1:02:11<02:43,  5.01it/s]
 92%|█████████▏| 9181/10000 [1:02:11<02:45,  4.95it/s]
 92%|█████████▏| 9182/10000 [1:02:11<02:44,  4.96it/s]
 92%|█████████▏| 9183/10000 [1:02:11<02:45,  4.95it/s]
 92%|█████████▏| 9184/10000 [1:02:11<02:45,  4.94it/s]
 92%|█████████▏| 9185/10000 [1:02:12<02:46,  4.89it/s]
 92%|█████████▏| 9186/10000 [1:02:12<02:48,  4.83it/s]
 92%|█████████▏| 9187/10000 [1:02:12<02:46,  4.88it/s]
 92%|█████████▏| 9188/10000 [1:02:12<02:44,  4.94it/s]
 92%|█████████▏| 9189/10000 [1:02:12<02:43,  4.97it/s]
 92%|█████████▏| 9190/10000 [1:02:13<02:41,  5.03it/s]
                                                      
{'loss': 0.0705, 'grad_norm': 0.6172126531600952, 'learning_rate': 1.787432092061475e-06}

 92%|█████████▏| 9190/10000 [1:02:13<02:41,  5.03it/s]
 92%|█████████▏| 9191/10000 [1:02:13<02:44,  4.93it/s]
 92%|█████████▏| 9192/10000 [1:02:13<02:41,  5.00it/s]
 92%|█████████▏| 9193/10000 [1:02:13<02:38,  5.10it/s]
 92%|█████████▏| 9194/10000 [1:02:13<02:36,  5.16it/s]
 92%|█████████▏| 9195/10000 [1:02:13<02:35,  5.16it/s]
 92%|█████████▏| 9196/10000 [1:02:14<02:36,  5.12it/s]
 92%|█████████▏| 9197/10000 [1:02:14<02:38,  5.06it/s]
 92%|█████████▏| 9198/10000 [1:02:14<02:39,  5.03it/s]
 92%|█████████▏| 9199/10000 [1:02:14<02:40,  4.99it/s]
 92%|█████████▏| 9200/10000 [1:02:14<02:39,  5.01it/s]
                                                      
{'loss': 0.0626, 'grad_norm': 0.5759767889976501, 'learning_rate': 1.743880616801602e-06}

 92%|█████████▏| 9200/10000 [1:02:15<02:39,  5.01it/s]
 92%|█████████▏| 9201/10000 [1:02:15<02:42,  4.93it/s]
 92%|█████████▏| 9202/10000 [1:02:15<02:40,  4.97it/s]
 92%|█████████▏| 9203/10000 [1:02:15<02:39,  4.99it/s]
 92%|█████████▏| 9204/10000 [1:02:15<02:38,  5.02it/s]
 92%|█████████▏| 9205/10000 [1:02:15<02:38,  5.03it/s]
 92%|█████████▏| 9206/10000 [1:02:16<02:35,  5.11it/s]
 92%|█████████▏| 9207/10000 [1:02:16<02:34,  5.13it/s]
 92%|█████████▏| 9208/10000 [1:02:16<02:38,  5.00it/s]
 92%|█████████▏| 9209/10000 [1:02:16<02:37,  5.03it/s]
 92%|█████████▏| 9210/10000 [1:02:16<02:37,  5.03it/s]
                                                      
{'loss': 0.064, 'grad_norm': 0.6222781538963318, 'learning_rate': 1.7008568627319865e-06}

 92%|█████████▏| 9210/10000 [1:02:17<02:37,  5.03it/s]
 92%|█████████▏| 9211/10000 [1:02:17<02:39,  4.96it/s]
 92%|█████████▏| 9212/10000 [1:02:17<02:39,  4.93it/s]
 92%|█████████▏| 9213/10000 [1:02:17<02:39,  4.92it/s]Rank 0, Worker 3: Wait for shard 54 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

 92%|█████████▏| 9214/10000 [1:02:17<02:47,  4.70it/s]
 92%|█████████▏| 9215/10000 [1:02:18<02:46,  4.71it/s]
 92%|█████████▏| 9216/10000 [1:02:18<02:44,  4.77it/s]
 92%|█████████▏| 9217/10000 [1:02:18<02:57,  4.42it/s]
 92%|█████████▏| 9218/10000 [1:02:18<02:47,  4.67it/s]
 92%|█████████▏| 9219/10000 [1:02:18<02:43,  4.78it/s]
 92%|█████████▏| 9220/10000 [1:02:19<02:38,  4.93it/s]
                                                      
{'loss': 0.0639, 'grad_norm': 0.5419633984565735, 'learning_rate': 1.6583613003535226e-06}

 92%|█████████▏| 9220/10000 [1:02:19<02:38,  4.93it/s]
 92%|█████████▏| 9221/10000 [1:02:19<02:36,  4.99it/s]
 92%|█████████▏| 9222/10000 [1:02:19<02:36,  4.98it/s]
 92%|█████████▏| 9223/10000 [1:02:19<02:31,  5.13it/s]
 92%|█████████▏| 9224/10000 [1:02:19<02:30,  5.17it/s]
 92%|█████████▏| 9225/10000 [1:02:20<02:28,  5.21it/s]
 92%|█████████▏| 9226/10000 [1:02:20<02:25,  5.31it/s]
 92%|█████████▏| 9227/10000 [1:02:20<02:26,  5.29it/s]
 92%|█████████▏| 9228/10000 [1:02:20<02:25,  5.30it/s]
 92%|█████████▏| 9229/10000 [1:02:20<02:23,  5.35it/s]
 92%|█████████▏| 9230/10000 [1:02:20<02:24,  5.33it/s]
                                                      
{'loss': 0.0651, 'grad_norm': 0.6539579629898071, 'learning_rate': 1.6163943943908522e-06}

 92%|█████████▏| 9230/10000 [1:02:21<02:24,  5.33it/s]
 92%|█████████▏| 9231/10000 [1:02:21<02:31,  5.07it/s]
 92%|█████████▏| 9232/10000 [1:02:21<02:33,  4.99it/s]
 92%|█████████▏| 9233/10000 [1:02:21<02:39,  4.79it/s]
 92%|█████████▏| 9234/10000 [1:02:21<02:44,  4.65it/s]
 92%|█████████▏| 9235/10000 [1:02:22<02:45,  4.62it/s]
 92%|█████████▏| 9236/10000 [1:02:22<02:43,  4.68it/s]
 92%|█████████▏| 9237/10000 [1:02:22<02:34,  4.93it/s]
 92%|█████████▏| 9238/10000 [1:02:22<02:31,  5.02it/s]
 92%|█████████▏| 9239/10000 [1:02:22<02:28,  5.11it/s]
 92%|█████████▏| 9240/10000 [1:02:23<02:26,  5.19it/s]
                                                      
{'loss': 0.0629, 'grad_norm': 0.5587801337242126, 'learning_rate': 1.5749566037873476e-06}

 92%|█████████▏| 9240/10000 [1:02:23<02:26,  5.19it/s]
 92%|█████████▏| 9241/10000 [1:02:23<02:26,  5.17it/s]
 92%|█████████▏| 9242/10000 [1:02:23<02:25,  5.19it/s]
 92%|█████████▏| 9243/10000 [1:02:23<02:23,  5.29it/s]
 92%|█████████▏| 9244/10000 [1:02:23<02:21,  5.36it/s]
 92%|█████████▏| 9245/10000 [1:02:23<02:19,  5.40it/s]
 92%|█████████▏| 9246/10000 [1:02:24<02:20,  5.38it/s]
 92%|█████████▏| 9247/10000 [1:02:24<02:18,  5.45it/s]
 92%|█████████▏| 9248/10000 [1:02:24<02:20,  5.36it/s]
 92%|█████████▏| 9249/10000 [1:02:24<02:20,  5.34it/s]
 92%|█████████▎| 9250/10000 [1:02:24<02:19,  5.39it/s]Rank 0, Worker 0: Caching shard...Rank 0, Worker 1: Caching shard...Rank 0, Worker 2: Caching shard...


Rank 0, Worker 4: Caching shard...Rank 0, Worker 3: Caching shard...

Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 15.91 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.36 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.37 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.64 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.73 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.78 seconds
Rank 0, Worker 0: Caching shard...
[held-out-eval] step 9250: eval/loss = 0.0829 (35s)

                                                      
{'loss': 0.0623, 'grad_norm': 0.707502543926239, 'learning_rate': 1.5340483817000428e-06}

 92%|█████████▎| 9250/10000 [1:03:00<02:19,  5.39it/s]
 93%|█████████▎| 9251/10000 [1:03:00<2:14:38, 10.79s/it]
 93%|█████████▎| 9252/10000 [1:03:00<1:34:55,  7.61s/it]
 93%|█████████▎| 9253/10000 [1:03:00<1:07:08,  5.39s/it]
 93%|█████████▎| 9254/10000 [1:03:01<47:43,  3.84s/it]  
 93%|█████████▎| 9255/10000 [1:03:01<34:00,  2.74s/it]
 93%|█████████▎| 9256/10000 [1:03:01<24:30,  1.98s/it]
 93%|█████████▎| 9257/10000 [1:03:01<17:54,  1.45s/it]
 93%|█████████▎| 9258/10000 [1:03:01<13:17,  1.07s/it]
 93%|█████████▎| 9259/10000 [1:03:02<10:06,  1.22it/s]
 93%|█████████▎| 9260/10000 [1:03:02<07:51,  1.57it/s]
                                                      
{'loss': 0.0703, 'grad_norm': 0.5606984496116638, 'learning_rate': 1.4936701754947101e-06}

 93%|█████████▎| 9260/10000 [1:03:02<07:51,  1.57it/s]
 93%|█████████▎| 9261/10000 [1:03:02<06:15,  1.97it/s]
 93%|█████████▎| 9262/10000 [1:03:02<05:01,  2.45it/s]
 93%|█████████▎| 9263/10000 [1:03:02<04:10,  2.94it/s]
 93%|█████████▎| 9264/10000 [1:03:02<03:33,  3.45it/s]
 93%|█████████▎| 9265/10000 [1:03:03<03:10,  3.86it/s]
 93%|█████████▎| 9266/10000 [1:03:03<02:55,  4.19it/s]
 93%|█████████▎| 9267/10000 [1:03:03<02:43,  4.49it/s]
 93%|█████████▎| 9268/10000 [1:03:03<02:35,  4.70it/s]
 93%|█████████▎| 9269/10000 [1:03:03<02:29,  4.90it/s]
 93%|█████████▎| 9270/10000 [1:03:04<02:25,  5.03it/s]
                                                      
{'loss': 0.0724, 'grad_norm': 0.5581846237182617, 'learning_rate': 1.4538224267409361e-06}

 93%|█████████▎| 9270/10000 [1:03:04<02:25,  5.03it/s]
 93%|█████████▎| 9271/10000 [1:03:04<02:25,  5.00it/s]
 93%|█████████▎| 9272/10000 [1:03:04<02:36,  4.65it/s]
 93%|█████████▎| 9273/10000 [1:03:04<02:33,  4.75it/s]
 93%|█████████▎| 9274/10000 [1:03:04<02:31,  4.81it/s]
 93%|█████████▎| 9275/10000 [1:03:05<02:30,  4.83it/s]
 93%|█████████▎| 9276/10000 [1:03:05<02:29,  4.86it/s]
 93%|█████████▎| 9277/10000 [1:03:05<02:26,  4.95it/s]
 93%|█████████▎| 9278/10000 [1:03:05<02:24,  4.99it/s]
 93%|█████████▎| 9279/10000 [1:03:05<02:23,  5.02it/s]
 93%|█████████▎| 9280/10000 [1:03:06<02:23,  5.00it/s]
                                                      
{'loss': 0.0774, 'grad_norm': 0.6549857258796692, 'learning_rate': 1.414505571207314e-06}

 93%|█████████▎| 9280/10000 [1:03:06<02:23,  5.00it/s]
 93%|█████████▎| 9281/10000 [1:03:06<02:25,  4.95it/s]Rank 0, Worker 5: Wait for shard 16 in dataset 0 in 0.00 seconds
Rank 0, Worker 5: Caching shard...

 93%|█████████▎| 9282/10000 [1:03:06<02:26,  4.89it/s]
 93%|█████████▎| 9283/10000 [1:03:06<02:26,  4.90it/s]
 93%|█████████▎| 9284/10000 [1:03:06<02:27,  4.84it/s]
 93%|█████████▎| 9285/10000 [1:03:07<02:33,  4.66it/s]
 93%|█████████▎| 9286/10000 [1:03:07<02:35,  4.59it/s]
 93%|█████████▎| 9287/10000 [1:03:07<02:31,  4.70it/s]
 93%|█████████▎| 9288/10000 [1:03:07<02:30,  4.74it/s]
 93%|█████████▎| 9289/10000 [1:03:08<02:27,  4.81it/s]
 93%|█████████▎| 9290/10000 [1:03:08<02:27,  4.82it/s]
                                                      
{'loss': 0.0611, 'grad_norm': 0.526965856552124, 'learning_rate': 1.3757200388566816e-06}

 93%|█████████▎| 9290/10000 [1:03:08<02:27,  4.82it/s]
 93%|█████████▎| 9291/10000 [1:03:08<02:30,  4.70it/s]
 93%|█████████▎| 9292/10000 [1:03:08<02:28,  4.77it/s]
 93%|█████████▎| 9293/10000 [1:03:08<02:29,  4.72it/s]
 93%|█████████▎| 9294/10000 [1:03:09<02:28,  4.75it/s]
 93%|█████████▎| 9295/10000 [1:03:09<02:23,  4.92it/s]
 93%|█████████▎| 9296/10000 [1:03:09<02:20,  5.00it/s]
 93%|█████████▎| 9297/10000 [1:03:09<02:19,  5.04it/s]
 93%|█████████▎| 9298/10000 [1:03:09<02:21,  4.98it/s]
 93%|█████████▎| 9299/10000 [1:03:10<02:22,  4.93it/s]
 93%|█████████▎| 9300/10000 [1:03:10<02:26,  4.79it/s]
                                                      
{'loss': 0.0617, 'grad_norm': 0.6523736715316772, 'learning_rate': 1.3374662538414074e-06}

 93%|█████████▎| 9300/10000 [1:03:10<02:26,  4.79it/s]
 93%|█████████▎| 9301/10000 [1:03:10<02:26,  4.77it/s]
 93%|█████████▎| 9302/10000 [1:03:10<02:24,  4.84it/s]
 93%|█████████▎| 9303/10000 [1:03:10<02:21,  4.92it/s]
 93%|█████████▎| 9304/10000 [1:03:11<02:18,  5.02it/s]
 93%|█████████▎| 9305/10000 [1:03:11<02:16,  5.10it/s]
 93%|█████████▎| 9306/10000 [1:03:11<02:16,  5.08it/s]
 93%|█████████▎| 9307/10000 [1:03:11<02:20,  4.92it/s]
 93%|█████████▎| 9308/10000 [1:03:11<02:25,  4.77it/s]
 93%|█████████▎| 9309/10000 [1:03:12<02:25,  4.74it/s]
 93%|█████████▎| 9310/10000 [1:03:12<02:23,  4.81it/s]
                                                      
{'loss': 0.0665, 'grad_norm': 0.6115444898605347, 'learning_rate': 1.2997446344987617e-06}

 93%|█████████▎| 9310/10000 [1:03:12<02:23,  4.81it/s]
 93%|█████████▎| 9311/10000 [1:03:12<02:24,  4.76it/s]
 93%|█████████▎| 9312/10000 [1:03:12<02:21,  4.87it/s]
 93%|█████████▎| 9313/10000 [1:03:12<02:19,  4.91it/s]
 93%|█████████▎| 9314/10000 [1:03:13<02:16,  5.02it/s]
 93%|█████████▎| 9315/10000 [1:03:13<02:13,  5.11it/s]
 93%|█████████▎| 9316/10000 [1:03:13<02:15,  5.06it/s]
 93%|█████████▎| 9317/10000 [1:03:13<02:14,  5.06it/s]
 93%|█████████▎| 9318/10000 [1:03:13<02:14,  5.07it/s]Rank 0, Worker 0: Wait for shard 42 in dataset 0 in 0.00 seconds
Rank 0, Worker 0: Caching shard...

 93%|█████████▎| 9319/10000 [1:03:14<02:16,  5.00it/s]
 93%|█████████▎| 9320/10000 [1:03:14<02:20,  4.85it/s]
                                                      
{'loss': 0.0602, 'grad_norm': 0.605048418045044, 'learning_rate': 1.262555593346315e-06}

 93%|█████████▎| 9320/10000 [1:03:14<02:20,  4.85it/s]
 93%|█████████▎| 9321/10000 [1:03:14<02:21,  4.80it/s]
 93%|█████████▎| 9322/10000 [1:03:14<02:20,  4.83it/s]
 93%|█████████▎| 9323/10000 [1:03:15<02:19,  4.84it/s]
 93%|█████████▎| 9324/10000 [1:03:15<02:17,  4.92it/s]
 93%|█████████▎| 9325/10000 [1:03:15<02:16,  4.94it/s]
 93%|█████████▎| 9326/10000 [1:03:15<02:15,  4.98it/s]
 93%|█████████▎| 9327/10000 [1:03:15<02:14,  5.02it/s]
 93%|█████████▎| 9328/10000 [1:03:15<02:14,  4.99it/s]
 93%|█████████▎| 9329/10000 [1:03:16<02:18,  4.85it/s]
 93%|█████████▎| 9330/10000 [1:03:16<02:22,  4.72it/s]
                                                      
{'loss': 0.0771, 'grad_norm': 0.48388490080833435, 'learning_rate': 1.2258995370774685e-06}

 93%|█████████▎| 9330/10000 [1:03:16<02:22,  4.72it/s]
 93%|█████████▎| 9331/10000 [1:03:16<02:24,  4.63it/s]
 93%|█████████▎| 9332/10000 [1:03:16<02:23,  4.67it/s]
 93%|█████████▎| 9333/10000 [1:03:17<02:18,  4.82it/s]
 93%|█████████▎| 9334/10000 [1:03:17<02:14,  4.96it/s]
 93%|█████████▎| 9335/10000 [1:03:17<02:15,  4.91it/s]
 93%|█████████▎| 9336/10000 [1:03:17<02:14,  4.92it/s]
 93%|█████████▎| 9337/10000 [1:03:17<02:16,  4.85it/s]
 93%|█████████▎| 9338/10000 [1:03:18<02:16,  4.85it/s]
 93%|█████████▎| 9339/10000 [1:03:18<02:15,  4.87it/s]
 93%|█████████▎| 9340/10000 [1:03:18<02:14,  4.90it/s]
                                                      
{'loss': 0.0732, 'grad_norm': 0.6403599977493286, 'learning_rate': 1.1897768665569798e-06}

 93%|█████████▎| 9340/10000 [1:03:18<02:14,  4.90it/s]
 93%|█████████▎| 9341/10000 [1:03:18<02:15,  4.87it/s]
 93%|█████████▎| 9342/10000 [1:03:18<02:10,  5.03it/s]
 93%|█████████▎| 9343/10000 [1:03:19<02:10,  5.02it/s]
 93%|█████████▎| 9344/10000 [1:03:19<02:10,  5.03it/s]
 93%|█████████▎| 9345/10000 [1:03:19<02:10,  5.01it/s]
 93%|█████████▎| 9346/10000 [1:03:19<02:11,  4.97it/s]
 93%|█████████▎| 9347/10000 [1:03:19<02:12,  4.92it/s]
 93%|█████████▎| 9348/10000 [1:03:20<02:13,  4.87it/s]
 93%|█████████▎| 9349/10000 [1:03:20<02:12,  4.93it/s]
 94%|█████████▎| 9350/10000 [1:03:20<02:11,  4.93it/s]
                                                      
{'loss': 0.0628, 'grad_norm': 0.6139552593231201, 'learning_rate': 1.1541879768165954e-06}

 94%|█████████▎| 9350/10000 [1:03:20<02:11,  4.93it/s]
 94%|█████████▎| 9351/10000 [1:03:20<02:12,  4.88it/s]
 94%|█████████▎| 9352/10000 [1:03:20<02:07,  5.08it/s]
 94%|█████████▎| 9353/10000 [1:03:21<02:04,  5.18it/s]
 94%|█████████▎| 9354/10000 [1:03:21<02:03,  5.23it/s]
 94%|█████████▎| 9355/10000 [1:03:21<02:03,  5.21it/s]
 94%|█████████▎| 9356/10000 [1:03:21<02:04,  5.16it/s]
 94%|█████████▎| 9357/10000 [1:03:21<02:05,  5.12it/s]
 94%|█████████▎| 9358/10000 [1:03:22<02:07,  5.02it/s]
 94%|█████████▎| 9359/10000 [1:03:22<02:07,  5.02it/s]
 94%|█████████▎| 9360/10000 [1:03:22<02:08,  4.97it/s]
                                                      
{'loss': 0.0674, 'grad_norm': 0.5721161961555481, 'learning_rate': 1.1191332570507085e-06}

 94%|█████████▎| 9360/10000 [1:03:22<02:08,  4.97it/s]
 94%|█████████▎| 9361/10000 [1:03:22<02:08,  4.99it/s]
 94%|█████████▎| 9362/10000 [1:03:22<02:05,  5.09it/s]
 94%|█████████▎| 9363/10000 [1:03:23<02:03,  5.15it/s]
 94%|█████████▎| 9364/10000 [1:03:23<02:02,  5.19it/s]
 94%|█████████▎| 9365/10000 [1:03:23<02:02,  5.18it/s]
 94%|█████████▎| 9366/10000 [1:03:23<02:02,  5.16it/s]
 94%|█████████▎| 9367/10000 [1:03:23<01:59,  5.29it/s]
 94%|█████████▎| 9368/10000 [1:03:23<01:58,  5.35it/s]
 94%|█████████▎| 9369/10000 [1:03:24<01:58,  5.33it/s]
 94%|█████████▎| 9370/10000 [1:03:24<01:59,  5.26it/s]
                                                      
{'loss': 0.0754, 'grad_norm': 0.5582464933395386, 'learning_rate': 1.0846130906121132e-06}

 94%|█████████▎| 9370/10000 [1:03:24<01:59,  5.26it/s]
 94%|█████████▎| 9371/10000 [1:03:24<02:02,  5.14it/s]
 94%|█████████▎| 9372/10000 [1:03:24<02:02,  5.11it/s]
 94%|█████████▎| 9373/10000 [1:03:24<02:01,  5.18it/s]
 94%|█████████▎| 9374/10000 [1:03:25<02:00,  5.19it/s]
 94%|█████████▍| 9375/10000 [1:03:25<01:57,  5.32it/s]
 94%|█████████▍| 9376/10000 [1:03:25<01:57,  5.32it/s]
 94%|█████████▍| 9377/10000 [1:03:25<01:56,  5.34it/s]
 94%|█████████▍| 9378/10000 [1:03:25<01:56,  5.33it/s]
 94%|█████████▍| 9379/10000 [1:03:26<01:58,  5.26it/s]
 94%|█████████▍| 9380/10000 [1:03:26<01:59,  5.20it/s]
                                                      
{'loss': 0.0718, 'grad_norm': 0.5939518213272095, 'learning_rate': 1.0506278550078131e-06}

 94%|█████████▍| 9380/10000 [1:03:26<01:59,  5.20it/s]
 94%|█████████▍| 9381/10000 [1:03:26<02:02,  5.07it/s]
 94%|█████████▍| 9382/10000 [1:03:26<02:02,  5.05it/s]
 94%|█████████▍| 9383/10000 [1:03:26<02:04,  4.96it/s]
 94%|█████████▍| 9384/10000 [1:03:27<02:01,  5.05it/s]
 94%|█████████▍| 9385/10000 [1:03:27<02:00,  5.10it/s]
 94%|█████████▍| 9386/10000 [1:03:27<02:05,  4.89it/s]
 94%|█████████▍| 9387/10000 [1:03:27<02:14,  4.55it/s]
 94%|█████████▍| 9388/10000 [1:03:27<02:17,  4.44it/s]
 94%|█████████▍| 9389/10000 [1:03:28<02:20,  4.36it/s]
 94%|█████████▍| 9390/10000 [1:03:28<02:21,  4.31it/s]
                                                      
{'loss': 0.0615, 'grad_norm': 0.6783204078674316, 'learning_rate': 1.0171779218949185e-06}

 94%|█████████▍| 9390/10000 [1:03:28<02:21,  4.31it/s]
 94%|█████████▍| 9391/10000 [1:03:28<02:18,  4.40it/s]
 94%|█████████▍| 9392/10000 [1:03:28<02:12,  4.59it/s]
 94%|█████████▍| 9393/10000 [1:03:29<02:07,  4.76it/s]
 94%|█████████▍| 9394/10000 [1:03:29<02:01,  4.98it/s]
 94%|█████████▍| 9395/10000 [1:03:29<02:00,  5.03it/s]
 94%|█████████▍| 9396/10000 [1:03:29<02:07,  4.73it/s]
 94%|█████████▍| 9397/10000 [1:03:29<02:11,  4.58it/s]
 94%|█████████▍| 9398/10000 [1:03:30<02:11,  4.57it/s]
 94%|█████████▍| 9399/10000 [1:03:30<02:11,  4.57it/s]
 94%|█████████▍| 9400/10000 [1:03:30<02:08,  4.67it/s]
                                                      
{'loss': 0.0613, 'grad_norm': 0.5898619294166565, 'learning_rate': 9.842636570765174e-07}

 94%|█████████▍| 9400/10000 [1:03:30<02:08,  4.67it/s]
 94%|█████████▍| 9401/10000 [1:03:30<02:07,  4.69it/s]
 94%|█████████▍| 9402/10000 [1:03:30<02:04,  4.79it/s]
 94%|█████████▍| 9403/10000 [1:03:31<02:01,  4.93it/s]
 94%|█████████▍| 9404/10000 [1:03:31<01:56,  5.11it/s]
 94%|█████████▍| 9405/10000 [1:03:31<01:56,  5.13it/s]
 94%|█████████▍| 9406/10000 [1:03:31<02:00,  4.94it/s]
 94%|█████████▍| 9407/10000 [1:03:31<02:05,  4.73it/s]
 94%|█████████▍| 9408/10000 [1:03:32<02:09,  4.58it/s]
 94%|█████████▍| 9409/10000 [1:03:32<02:11,  4.48it/s]
 94%|█████████▍| 9410/10000 [1:03:32<02:10,  4.54it/s]
                                                      
{'loss': 0.0661, 'grad_norm': 0.5533653497695923, 'learning_rate': 9.518854204977612e-07}

 94%|█████████▍| 9410/10000 [1:03:32<02:10,  4.54it/s]
 94%|█████████▍| 9411/10000 [1:03:32<02:09,  4.55it/s]
 94%|█████████▍| 9412/10000 [1:03:33<02:06,  4.66it/s]
 94%|█████████▍| 9413/10000 [1:03:33<02:04,  4.73it/s]
 94%|█████████▍| 9414/10000 [1:03:33<02:03,  4.76it/s]
 94%|█████████▍| 9415/10000 [1:03:33<02:07,  4.61it/s]
 94%|█████████▍| 9416/10000 [1:03:33<02:12,  4.42it/s]
 94%|█████████▍| 9417/10000 [1:03:34<02:24,  4.04it/s]
 94%|█████████▍| 9418/10000 [1:03:34<02:18,  4.21it/s]
 94%|█████████▍| 9419/10000 [1:03:34<02:14,  4.33it/s]
 94%|█████████▍| 9420/10000 [1:03:34<02:05,  4.62it/s]
                                                      
{'loss': 0.0622, 'grad_norm': 0.612332820892334, 'learning_rate': 9.200435662418349e-07}

 94%|█████████▍| 9420/10000 [1:03:34<02:05,  4.62it/s]
 94%|█████████▍| 9421/10000 [1:03:35<02:04,  4.66it/s]
 94%|█████████▍| 9422/10000 [1:03:35<02:05,  4.62it/s]
 94%|█████████▍| 9423/10000 [1:03:35<02:08,  4.51it/s]
 94%|█████████▍| 9424/10000 [1:03:35<02:13,  4.32it/s]
 94%|█████████▍| 9425/10000 [1:03:36<02:20,  4.10it/s]
 94%|█████████▍| 9426/10000 [1:03:36<02:18,  4.14it/s]
 94%|█████████▍| 9427/10000 [1:03:36<02:13,  4.29it/s]
 94%|█████████▍| 9428/10000 [1:03:36<02:06,  4.51it/s]
 94%|█████████▍| 9429/10000 [1:03:36<02:01,  4.71it/s]
 94%|█████████▍| 9430/10000 [1:03:37<02:00,  4.73it/s]
                                                      
{'loss': 0.0819, 'grad_norm': 0.6349133253097534, 'learning_rate': 8.887384425261658e-07}

 94%|█████████▍| 9430/10000 [1:03:37<02:00,  4.73it/s]
 94%|█████████▍| 9431/10000 [1:03:37<02:04,  4.56it/s]
 94%|█████████▍| 9432/10000 [1:03:37<02:07,  4.47it/s]
 94%|█████████▍| 9433/10000 [1:03:37<02:14,  4.22it/s]
 94%|█████████▍| 9434/10000 [1:03:38<02:13,  4.25it/s]
 94%|█████████▍| 9435/10000 [1:03:38<02:09,  4.35it/s]
 94%|█████████▍| 9436/10000 [1:03:38<02:06,  4.46it/s]
 94%|█████████▍| 9437/10000 [1:03:38<02:02,  4.58it/s]
 94%|█████████▍| 9438/10000 [1:03:38<02:00,  4.67it/s]
 94%|█████████▍| 9439/10000 [1:03:39<01:58,  4.74it/s]
 94%|█████████▍| 9440/10000 [1:03:39<02:03,  4.55it/s]
                                                      
{'loss': 0.0634, 'grad_norm': 0.5273129343986511, 'learning_rate': 8.579703916985648e-07}

 94%|█████████▍| 9440/10000 [1:03:39<02:03,  4.55it/s]
 94%|█████████▍| 9441/10000 [1:03:39<02:14,  4.17it/s]
 94%|█████████▍| 9442/10000 [1:03:39<02:15,  4.13it/s]
 94%|█████████▍| 9443/10000 [1:03:40<02:13,  4.19it/s]
 94%|█████████▍| 9444/10000 [1:03:40<02:05,  4.42it/s]
 94%|█████████▍| 9445/10000 [1:03:40<01:58,  4.68it/s]
 94%|█████████▍| 9446/10000 [1:03:40<01:53,  4.87it/s]
 94%|█████████▍| 9447/10000 [1:03:40<01:54,  4.82it/s]
 94%|█████████▍| 9448/10000 [1:03:41<01:58,  4.65it/s]
 94%|█████████▍| 9449/10000 [1:03:41<02:01,  4.52it/s]
 94%|█████████▍| 9450/10000 [1:03:41<02:02,  4.49it/s]
                                                      
{'loss': 0.0708, 'grad_norm': 0.515934944152832, 'learning_rate': 8.277397502335194e-07}

 94%|█████████▍| 9450/10000 [1:03:41<02:02,  4.49it/s]
 95%|█████████▍| 9451/10000 [1:03:41<02:02,  4.46it/s]
 95%|█████████▍| 9452/10000 [1:03:42<01:58,  4.63it/s]
 95%|█████████▍| 9453/10000 [1:03:42<01:55,  4.74it/s]
 95%|█████████▍| 9454/10000 [1:03:42<01:54,  4.76it/s]
 95%|█████████▍| 9455/10000 [1:03:42<01:53,  4.82it/s]
 95%|█████████▍| 9456/10000 [1:03:42<01:53,  4.79it/s]
 95%|█████████▍| 9457/10000 [1:03:43<01:58,  4.59it/s]
 95%|█████████▍| 9458/10000 [1:03:43<01:57,  4.60it/s]
 95%|█████████▍| 9459/10000 [1:03:43<01:56,  4.63it/s]
 95%|█████████▍| 9460/10000 [1:03:43<01:56,  4.63it/s]
                                                      
{'loss': 0.0669, 'grad_norm': 0.517819344997406, 'learning_rate': 7.980468487284675e-07}

 95%|█████████▍| 9460/10000 [1:03:43<01:56,  4.63it/s]
 95%|█████████▍| 9461/10000 [1:03:43<01:58,  4.55it/s]
 95%|█████████▍| 9462/10000 [1:03:44<01:54,  4.69it/s]
 95%|█████████▍| 9463/10000 [1:03:44<01:51,  4.82it/s]
 95%|█████████▍| 9464/10000 [1:03:44<01:51,  4.83it/s]
 95%|█████████▍| 9465/10000 [1:03:44<01:54,  4.68it/s]
 95%|█████████▍| 9466/10000 [1:03:45<01:54,  4.67it/s]
 95%|█████████▍| 9467/10000 [1:03:45<01:58,  4.50it/s]
 95%|█████████▍| 9468/10000 [1:03:45<02:02,  4.35it/s]
 95%|█████████▍| 9469/10000 [1:03:45<01:58,  4.47it/s]
 95%|█████████▍| 9470/10000 [1:03:45<01:58,  4.47it/s]
                                                      
{'loss': 0.0638, 'grad_norm': 0.5353856086730957, 'learning_rate': 7.688920119002297e-07}

 95%|█████████▍| 9470/10000 [1:03:45<01:58,  4.47it/s]
 95%|█████████▍| 9471/10000 [1:03:46<01:53,  4.64it/s]
 95%|█████████▍| 9472/10000 [1:03:46<01:49,  4.83it/s]
 95%|█████████▍| 9473/10000 [1:03:46<01:48,  4.84it/s]
 95%|█████████▍| 9474/10000 [1:03:46<01:51,  4.73it/s]
 95%|█████████▍| 9475/10000 [1:03:46<01:51,  4.71it/s]
 95%|█████████▍| 9476/10000 [1:03:47<01:52,  4.67it/s]
 95%|█████████▍| 9477/10000 [1:03:47<01:52,  4.63it/s]
 95%|█████████▍| 9478/10000 [1:03:47<01:53,  4.61it/s]
 95%|█████████▍| 9479/10000 [1:03:47<01:53,  4.59it/s]
 95%|█████████▍| 9480/10000 [1:03:48<01:50,  4.70it/s]
                                                      
{'loss': 0.0607, 'grad_norm': 0.5077387094497681, 'learning_rate': 7.402755585814269e-07}

 95%|█████████▍| 9480/10000 [1:03:48<01:50,  4.70it/s]
 95%|█████████▍| 9481/10000 [1:03:48<01:47,  4.83it/s]
 95%|█████████▍| 9482/10000 [1:03:48<01:43,  5.01it/s]
 95%|█████████▍| 9483/10000 [1:03:48<01:42,  5.04it/s]
 95%|█████████▍| 9484/10000 [1:03:48<01:44,  4.93it/s]
 95%|█████████▍| 9485/10000 [1:03:49<01:45,  4.86it/s]
 95%|█████████▍| 9486/10000 [1:03:49<01:46,  4.82it/s]
 95%|█████████▍| 9487/10000 [1:03:49<01:47,  4.75it/s]
 95%|█████████▍| 9488/10000 [1:03:49<01:46,  4.82it/s]
 95%|█████████▍| 9489/10000 [1:03:49<01:43,  4.93it/s]
 95%|█████████▍| 9490/10000 [1:03:50<01:42,  4.99it/s]
                                                      
{'loss': 0.0723, 'grad_norm': 0.5764630436897278, 'learning_rate': 7.121978017170073e-07}

 95%|█████████▍| 9490/10000 [1:03:50<01:42,  4.99it/s]
 95%|█████████▍| 9491/10000 [1:03:50<01:40,  5.09it/s]
 95%|█████████▍| 9492/10000 [1:03:50<01:38,  5.16it/s]
 95%|█████████▍| 9493/10000 [1:03:50<01:38,  5.13it/s]
 95%|█████████▍| 9494/10000 [1:03:50<01:40,  5.05it/s]
 95%|█████████▍| 9495/10000 [1:03:51<01:40,  5.01it/s]
 95%|█████████▍| 9496/10000 [1:03:51<01:45,  4.77it/s]
 95%|█████████▍| 9497/10000 [1:03:51<01:44,  4.83it/s]
 95%|█████████▍| 9498/10000 [1:03:51<01:44,  4.80it/s]
 95%|█████████▍| 9499/10000 [1:03:52<02:05,  4.00it/s]
 95%|█████████▌| 9500/10000 [1:03:52<01:57,  4.27it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 1: Caching shard...Rank 0, Worker 2: Caching shard...

Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.52 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.82 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.92 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 18.13 seconds
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 18.18 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 18.21 seconds
Rank 0, Worker 0: Caching shard...
[held-out-eval] step 9500: eval/loss = 0.0855 (35s)

                                                      
{'loss': 0.0665, 'grad_norm': 0.5937212109565735, 'learning_rate': 6.846590483608306e-07}

 95%|█████████▌| 9500/10000 [1:04:27<01:57,  4.27it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-9500/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-9500
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-9500/wandb_config.json

 95%|█████████▌| 9501/10000 [1:04:55<2:39:25, 19.17s/it]
 95%|█████████▌| 9502/10000 [1:04:55<1:51:56, 13.49s/it]
 95%|█████████▌| 9503/10000 [1:04:56<1:18:42,  9.50s/it]
 95%|█████████▌| 9504/10000 [1:04:56<55:28,  6.71s/it]  
 95%|█████████▌| 9505/10000 [1:04:56<39:16,  4.76s/it]
 95%|█████████▌| 9506/10000 [1:04:56<27:58,  3.40s/it]
 95%|█████████▌| 9507/10000 [1:04:56<20:03,  2.44s/it]
 95%|█████████▌| 9508/10000 [1:04:57<14:31,  1.77s/it]
 95%|█████████▌| 9509/10000 [1:04:57<10:43,  1.31s/it]
 95%|█████████▌| 9510/10000 [1:04:57<08:02,  1.01it/s]
                                                      
{'loss': 0.0953, 'grad_norm': 0.6795274615287781, 'learning_rate': 6.576595996722834e-07}

 95%|█████████▌| 9510/10000 [1:04:57<08:02,  1.01it/s]
 95%|█████████▌| 9511/10000 [1:04:57<06:10,  1.32it/s]
 95%|█████████▌| 9512/10000 [1:04:57<04:49,  1.68it/s]
 95%|█████████▌| 9513/10000 [1:04:58<03:53,  2.09it/s]
 95%|█████████▌| 9514/10000 [1:04:58<03:12,  2.53it/s]
 95%|█████████▌| 9515/10000 [1:04:58<02:43,  2.97it/s]
 95%|█████████▌| 9516/10000 [1:04:58<02:23,  3.38it/s]
 95%|█████████▌| 9517/10000 [1:04:58<02:09,  3.73it/s]
 95%|█████████▌| 9518/10000 [1:04:59<02:01,  3.98it/s]
 95%|█████████▌| 9519/10000 [1:04:59<01:56,  4.13it/s]
 95%|█████████▌| 9520/10000 [1:04:59<01:54,  4.19it/s]
                                                      
{'loss': 0.0572, 'grad_norm': 0.7077244520187378, 'learning_rate': 6.311997509130141e-07}

 95%|█████████▌| 9520/10000 [1:04:59<01:54,  4.19it/s]
 95%|█████████▌| 9521/10000 [1:04:59<01:54,  4.17it/s]
 95%|█████████▌| 9522/10000 [1:05:00<01:50,  4.31it/s]
 95%|█████████▌| 9523/10000 [1:05:00<01:46,  4.48it/s]
 95%|█████████▌| 9524/10000 [1:05:00<01:40,  4.72it/s]
 95%|█████████▌| 9525/10000 [1:05:00<01:39,  4.76it/s]
 95%|█████████▌| 9526/10000 [1:05:00<01:39,  4.77it/s]
 95%|█████████▌| 9527/10000 [1:05:01<01:43,  4.57it/s]
 95%|█████████▌| 9528/10000 [1:05:01<01:46,  4.44it/s]
 95%|█████████▌| 9529/10000 [1:05:01<01:49,  4.29it/s]
 95%|█████████▌| 9530/10000 [1:05:01<01:46,  4.39it/s]
                                                      
{'loss': 0.0696, 'grad_norm': 0.5086929202079773, 'learning_rate': 6.052797914436803e-07}

 95%|█████████▌| 9530/10000 [1:05:01<01:46,  4.39it/s]
 95%|█████████▌| 9531/10000 [1:05:02<01:44,  4.48it/s]
 95%|█████████▌| 9532/10000 [1:05:02<01:38,  4.73it/s]
 95%|█████████▌| 9533/10000 [1:05:02<01:43,  4.49it/s]
 95%|█████████▌| 9534/10000 [1:05:02<01:41,  4.61it/s]
 95%|█████████▌| 9535/10000 [1:05:02<01:40,  4.62it/s]
 95%|█████████▌| 9536/10000 [1:05:03<01:40,  4.61it/s]
 95%|█████████▌| 9537/10000 [1:05:03<01:44,  4.44it/s]
 95%|█████████▌| 9538/10000 [1:05:03<01:47,  4.29it/s]
 95%|█████████▌| 9539/10000 [1:05:03<01:53,  4.06it/s]
 95%|█████████▌| 9540/10000 [1:05:04<01:47,  4.29it/s]
                                                      
{'loss': 0.0491, 'grad_norm': 0.8160368800163269, 'learning_rate': 5.799000047208181e-07}

 95%|█████████▌| 9540/10000 [1:05:04<01:47,  4.29it/s]
 95%|█████████▌| 9541/10000 [1:05:04<01:42,  4.46it/s]
 95%|█████████▌| 9542/10000 [1:05:04<01:38,  4.63it/s]
 95%|█████████▌| 9543/10000 [1:05:04<01:38,  4.64it/s]
 95%|█████████▌| 9544/10000 [1:05:04<01:38,  4.61it/s]
 95%|█████████▌| 9545/10000 [1:05:05<01:41,  4.47it/s]
 95%|█████████▌| 9546/10000 [1:05:05<01:43,  4.39it/s]
 95%|█████████▌| 9547/10000 [1:05:05<01:43,  4.38it/s]
 95%|█████████▌| 9548/10000 [1:05:05<01:40,  4.50it/s]
 95%|█████████▌| 9549/10000 [1:05:06<01:36,  4.69it/s]
 96%|█████████▌| 9550/10000 [1:05:06<01:33,  4.81it/s]
                                                      
{'loss': 0.0878, 'grad_norm': 0.5942310690879822, 'learning_rate': 5.550606682937054e-07}

 96%|█████████▌| 9550/10000 [1:05:06<01:33,  4.81it/s]
 96%|█████████▌| 9551/10000 [1:05:06<01:34,  4.74it/s]
 96%|█████████▌| 9552/10000 [1:05:06<01:37,  4.58it/s]
 96%|█████████▌| 9553/10000 [1:05:06<01:37,  4.59it/s]
 96%|█████████▌| 9554/10000 [1:05:07<01:36,  4.63it/s]
 96%|█████████▌| 9555/10000 [1:05:07<01:35,  4.64it/s]
 96%|█████████▌| 9556/10000 [1:05:07<01:33,  4.76it/s]
 96%|█████████▌| 9557/10000 [1:05:07<01:30,  4.87it/s]
 96%|█████████▌| 9558/10000 [1:05:07<01:30,  4.91it/s]
 96%|█████████▌| 9559/10000 [1:05:08<01:29,  4.92it/s]
 96%|█████████▌| 9560/10000 [1:05:08<01:31,  4.78it/s]
                                                      
{'loss': 0.0573, 'grad_norm': 0.7997192144393921, 'learning_rate': 5.307620538013481e-07}

 96%|█████████▌| 9560/10000 [1:05:08<01:31,  4.78it/s]
 96%|█████████▌| 9561/10000 [1:05:08<01:32,  4.73it/s]
 96%|█████████▌| 9562/10000 [1:05:08<01:33,  4.68it/s]
 96%|█████████▌| 9563/10000 [1:05:09<01:34,  4.61it/s]
 96%|█████████▌| 9564/10000 [1:05:09<01:34,  4.59it/s]
 96%|█████████▌| 9565/10000 [1:05:09<01:36,  4.52it/s]
 96%|█████████▌| 9566/10000 [1:05:09<01:36,  4.48it/s]
 96%|█████████▌| 9567/10000 [1:05:09<01:33,  4.62it/s]
 96%|█████████▌| 9568/10000 [1:05:10<01:33,  4.60it/s]
 96%|█████████▌| 9569/10000 [1:05:10<01:31,  4.70it/s]
 96%|█████████▌| 9570/10000 [1:05:10<01:30,  4.74it/s]
                                                      
{'loss': 0.0618, 'grad_norm': 0.6010105013847351, 'learning_rate': 5.070044269694874e-07}

 96%|█████████▌| 9570/10000 [1:05:10<01:30,  4.74it/s]
 96%|█████████▌| 9571/10000 [1:05:10<01:34,  4.54it/s]
 96%|█████████▌| 9572/10000 [1:05:10<01:35,  4.48it/s]
 96%|█████████▌| 9573/10000 [1:05:11<01:37,  4.37it/s]
 96%|█████████▌| 9574/10000 [1:05:11<01:37,  4.37it/s]
 96%|█████████▌| 9575/10000 [1:05:11<01:33,  4.54it/s]
 96%|█████████▌| 9576/10000 [1:05:11<01:30,  4.70it/s]
 96%|█████████▌| 9577/10000 [1:05:12<01:27,  4.82it/s]
 96%|█████████▌| 9578/10000 [1:05:12<01:27,  4.83it/s]
 96%|█████████▌| 9579/10000 [1:05:12<01:28,  4.74it/s]
 96%|█████████▌| 9580/10000 [1:05:12<01:28,  4.76it/s]
                                                      
{'loss': 0.0583, 'grad_norm': 0.5644136667251587, 'learning_rate': 4.837880476077417e-07}

 96%|█████████▌| 9580/10000 [1:05:12<01:28,  4.76it/s]
 96%|█████████▌| 9581/10000 [1:05:12<01:30,  4.64it/s]
 96%|█████████▌| 9582/10000 [1:05:13<01:30,  4.63it/s]
 96%|█████████▌| 9583/10000 [1:05:13<01:30,  4.62it/s]
 96%|█████████▌| 9584/10000 [1:05:13<01:27,  4.74it/s]
 96%|█████████▌| 9585/10000 [1:05:13<01:24,  4.93it/s]
 96%|█████████▌| 9586/10000 [1:05:13<01:21,  5.09it/s]
 96%|█████████▌| 9587/10000 [1:05:14<01:20,  5.11it/s]
 96%|█████████▌| 9588/10000 [1:05:14<01:21,  5.06it/s]
 96%|█████████▌| 9589/10000 [1:05:14<01:23,  4.94it/s]
 96%|█████████▌| 9590/10000 [1:05:14<01:25,  4.79it/s]
                                                      
{'loss': 0.0742, 'grad_norm': 0.5064728856086731, 'learning_rate': 4.6111316960670835e-07}

 96%|█████████▌| 9590/10000 [1:05:14<01:25,  4.79it/s]
 96%|█████████▌| 9591/10000 [1:05:14<01:26,  4.74it/s]
 96%|█████████▌| 9592/10000 [1:05:15<01:26,  4.71it/s]
 96%|█████████▌| 9593/10000 [1:05:15<01:28,  4.61it/s]
 96%|█████████▌| 9594/10000 [1:05:15<01:28,  4.60it/s]
 96%|█████████▌| 9595/10000 [1:05:15<01:26,  4.67it/s]
 96%|█████████▌| 9596/10000 [1:05:16<01:26,  4.64it/s]
 96%|█████████▌| 9597/10000 [1:05:16<01:28,  4.54it/s]
 96%|█████████▌| 9598/10000 [1:05:16<01:29,  4.52it/s]
 96%|█████████▌| 9599/10000 [1:05:16<01:29,  4.47it/s]
 96%|█████████▌| 9600/10000 [1:05:16<01:32,  4.30it/s]
                                                      
{'loss': 0.0675, 'grad_norm': 0.739006519317627, 'learning_rate': 4.389800409352218e-07}

 96%|█████████▌| 9600/10000 [1:05:17<01:32,  4.30it/s]
 96%|█████████▌| 9601/10000 [1:05:17<01:36,  4.13it/s]
 96%|█████████▌| 9602/10000 [1:05:17<01:36,  4.13it/s]
 96%|█████████▌| 9603/10000 [1:05:17<01:33,  4.24it/s]
 96%|█████████▌| 9604/10000 [1:05:17<01:27,  4.55it/s]
 96%|█████████▌| 9605/10000 [1:05:18<01:23,  4.71it/s]
 96%|█████████▌| 9606/10000 [1:05:18<01:22,  4.78it/s]
 96%|█████████▌| 9607/10000 [1:05:18<01:21,  4.82it/s]
 96%|█████████▌| 9608/10000 [1:05:18<01:21,  4.82it/s]
 96%|█████████▌| 9609/10000 [1:05:18<01:20,  4.84it/s]
 96%|█████████▌| 9610/10000 [1:05:19<01:21,  4.76it/s]
                                                      
{'loss': 0.0671, 'grad_norm': 0.7611128687858582, 'learning_rate': 4.173889036376277e-07}

 96%|█████████▌| 9610/10000 [1:05:19<01:21,  4.76it/s]
 96%|█████████▌| 9611/10000 [1:05:19<01:23,  4.67it/s]
 96%|█████████▌| 9612/10000 [1:05:19<01:19,  4.91it/s]
 96%|█████████▌| 9613/10000 [1:05:19<01:17,  5.00it/s]
 96%|█████████▌| 9614/10000 [1:05:19<01:16,  5.02it/s]
 96%|█████████▌| 9615/10000 [1:05:20<01:17,  4.99it/s]
 96%|█████████▌| 9616/10000 [1:05:20<01:18,  4.87it/s]
 96%|█████████▌| 9617/10000 [1:05:20<01:19,  4.79it/s]
 96%|█████████▌| 9618/10000 [1:05:20<01:19,  4.82it/s]
 96%|█████████▌| 9619/10000 [1:05:20<01:17,  4.89it/s]
 96%|█████████▌| 9620/10000 [1:05:21<01:15,  5.01it/s]
                                                      
{'loss': 0.0531, 'grad_norm': 0.5092380046844482, 'learning_rate': 3.963399938311463e-07}

 96%|█████████▌| 9620/10000 [1:05:21<01:15,  5.01it/s]
 96%|█████████▌| 9621/10000 [1:05:21<01:14,  5.09it/s]
 96%|█████████▌| 9622/10000 [1:05:21<01:13,  5.14it/s]
 96%|█████████▌| 9623/10000 [1:05:21<01:12,  5.17it/s]
 96%|█████████▌| 9624/10000 [1:05:21<01:16,  4.90it/s]
 96%|█████████▋| 9625/10000 [1:05:22<01:18,  4.78it/s]
 96%|█████████▋| 9626/10000 [1:05:22<01:19,  4.68it/s]
 96%|█████████▋| 9627/10000 [1:05:22<01:21,  4.59it/s]
 96%|█████████▋| 9628/10000 [1:05:22<01:21,  4.56it/s]
 96%|█████████▋| 9629/10000 [1:05:23<01:20,  4.60it/s]
 96%|█████████▋| 9630/10000 [1:05:23<01:18,  4.72it/s]
                                                      
{'loss': 0.0571, 'grad_norm': 0.7125602960586548, 'learning_rate': 3.7583354170328545e-07}

 96%|█████████▋| 9630/10000 [1:05:23<01:18,  4.72it/s]
 96%|█████████▋| 9631/10000 [1:05:23<01:19,  4.65it/s]
 96%|█████████▋| 9632/10000 [1:05:23<01:18,  4.70it/s]
 96%|█████████▋| 9633/10000 [1:05:23<01:19,  4.62it/s]
 96%|█████████▋| 9634/10000 [1:05:24<01:18,  4.69it/s]
 96%|█████████▋| 9635/10000 [1:05:24<01:17,  4.72it/s]
 96%|█████████▋| 9636/10000 [1:05:24<01:19,  4.57it/s]
 96%|█████████▋| 9637/10000 [1:05:24<01:23,  4.35it/s]
 96%|█████████▋| 9638/10000 [1:05:25<01:27,  4.12it/s]
 96%|█████████▋| 9639/10000 [1:05:25<01:26,  4.16it/s]
 96%|█████████▋| 9640/10000 [1:05:25<01:26,  4.17it/s]
                                                      
{'loss': 0.0733, 'grad_norm': 0.8087239265441895, 'learning_rate': 3.558697715093207e-07}

 96%|█████████▋| 9640/10000 [1:05:25<01:26,  4.17it/s]
 96%|█████████▋| 9641/10000 [1:05:25<01:24,  4.26it/s]
 96%|█████████▋| 9642/10000 [1:05:25<01:20,  4.47it/s]
 96%|█████████▋| 9643/10000 [1:05:26<01:16,  4.67it/s]
 96%|█████████▋| 9644/10000 [1:05:26<01:14,  4.78it/s]
 96%|█████████▋| 9645/10000 [1:05:26<01:12,  4.87it/s]
 96%|█████████▋| 9646/10000 [1:05:26<01:13,  4.84it/s]
 96%|█████████▋| 9647/10000 [1:05:26<01:14,  4.76it/s]
 96%|█████████▋| 9648/10000 [1:05:27<01:13,  4.78it/s]
 96%|█████████▋| 9649/10000 [1:05:27<01:12,  4.81it/s]
 96%|█████████▋| 9650/10000 [1:05:27<01:12,  4.85it/s]
                                                      
{'loss': 0.0747, 'grad_norm': 0.6419918537139893, 'learning_rate': 3.3644890156983576e-07}

 96%|█████████▋| 9650/10000 [1:05:27<01:12,  4.85it/s]
 97%|█████████▋| 9651/10000 [1:05:27<01:11,  4.87it/s]
 97%|█████████▋| 9652/10000 [1:05:28<01:10,  4.94it/s]
 97%|█████████▋| 9653/10000 [1:05:28<01:10,  4.94it/s]
 97%|█████████▋| 9654/10000 [1:05:28<01:09,  4.98it/s]
 97%|█████████▋| 9655/10000 [1:05:28<01:07,  5.09it/s]
 97%|█████████▋| 9656/10000 [1:05:28<01:08,  5.02it/s]
 97%|█████████▋| 9657/10000 [1:05:29<01:09,  4.95it/s]
 97%|█████████▋| 9658/10000 [1:05:29<01:09,  4.95it/s]
 97%|█████████▋| 9659/10000 [1:05:29<01:09,  4.94it/s]
 97%|█████████▋| 9660/10000 [1:05:29<01:09,  4.86it/s]
                                                      
{'loss': 0.055, 'grad_norm': 0.5893635153770447, 'learning_rate': 3.175711442683638e-07}

 97%|█████████▋| 9660/10000 [1:05:29<01:09,  4.86it/s]
 97%|█████████▋| 9661/10000 [1:05:29<01:09,  4.90it/s]
 97%|█████████▋| 9662/10000 [1:05:30<01:07,  4.99it/s]
 97%|█████████▋| 9663/10000 [1:05:30<01:06,  5.04it/s]
 97%|█████████▋| 9664/10000 [1:05:30<01:06,  5.04it/s]
 97%|█████████▋| 9665/10000 [1:05:30<01:07,  4.98it/s]
 97%|█████████▋| 9666/10000 [1:05:30<01:07,  4.97it/s]
 97%|█████████▋| 9667/10000 [1:05:31<01:08,  4.87it/s]
 97%|█████████▋| 9668/10000 [1:05:31<01:08,  4.83it/s]
 97%|█████████▋| 9669/10000 [1:05:31<01:09,  4.78it/s]
 97%|█████████▋| 9670/10000 [1:05:31<01:08,  4.79it/s]
                                                      
{'loss': 0.0676, 'grad_norm': 0.5475870966911316, 'learning_rate': 2.9923670604902197e-07}

 97%|█████████▋| 9670/10000 [1:05:31<01:08,  4.79it/s]
 97%|█████████▋| 9671/10000 [1:05:31<01:08,  4.79it/s]
 97%|█████████▋| 9672/10000 [1:05:32<01:07,  4.83it/s]
 97%|█████████▋| 9673/10000 [1:05:32<01:07,  4.86it/s]
 97%|█████████▋| 9674/10000 [1:05:32<01:07,  4.81it/s]
 97%|█████████▋| 9675/10000 [1:05:32<01:08,  4.71it/s]
 97%|█████████▋| 9676/10000 [1:05:32<01:09,  4.65it/s]
 97%|█████████▋| 9677/10000 [1:05:33<01:10,  4.61it/s]
 97%|█████████▋| 9678/10000 [1:05:33<01:07,  4.74it/s]
 97%|█████████▋| 9679/10000 [1:05:33<01:04,  4.97it/s]
 97%|█████████▋| 9680/10000 [1:05:33<01:02,  5.13it/s]
                                                      
{'loss': 0.0808, 'grad_norm': 0.8972082734107971, 'learning_rate': 2.814457874143028e-07}

 97%|█████████▋| 9680/10000 [1:05:33<01:02,  5.13it/s]
 97%|█████████▋| 9681/10000 [1:05:33<01:04,  4.98it/s]
 97%|█████████▋| 9682/10000 [1:05:34<01:05,  4.89it/s]
 97%|█████████▋| 9683/10000 [1:05:34<01:05,  4.83it/s]
 97%|█████████▋| 9684/10000 [1:05:34<01:05,  4.79it/s]
 97%|█████████▋| 9685/10000 [1:05:34<01:05,  4.84it/s]
 97%|█████████▋| 9686/10000 [1:05:34<01:03,  4.92it/s]
 97%|█████████▋| 9687/10000 [1:05:35<01:02,  5.03it/s]
 97%|█████████▋| 9688/10000 [1:05:35<01:00,  5.15it/s]
 97%|█████████▋| 9689/10000 [1:05:35<00:59,  5.25it/s]
 97%|█████████▋| 9690/10000 [1:05:35<00:59,  5.22it/s]
                                                      
{'loss': 0.0676, 'grad_norm': 0.7143060564994812, 'learning_rate': 2.641985829228366e-07}

 97%|█████████▋| 9690/10000 [1:05:35<00:59,  5.22it/s]
 97%|█████████▋| 9691/10000 [1:05:35<01:02,  4.91it/s]
 97%|█████████▋| 9692/10000 [1:05:36<01:03,  4.86it/s]
 97%|█████████▋| 9693/10000 [1:05:36<01:05,  4.70it/s]
 97%|█████████▋| 9694/10000 [1:05:36<01:04,  4.78it/s]
 97%|█████████▋| 9695/10000 [1:05:36<01:02,  4.88it/s]
 97%|█████████▋| 9696/10000 [1:05:36<00:59,  5.08it/s]
 97%|█████████▋| 9697/10000 [1:05:37<00:57,  5.23it/s]
 97%|█████████▋| 9698/10000 [1:05:37<00:58,  5.15it/s]
 97%|█████████▋| 9699/10000 [1:05:37<01:00,  4.95it/s]
 97%|█████████▋| 9700/10000 [1:05:37<01:00,  4.95it/s]
                                                      
{'loss': 0.0743, 'grad_norm': 0.5767272114753723, 'learning_rate': 2.474952811872877e-07}

 97%|█████████▋| 9700/10000 [1:05:37<01:00,  4.95it/s]
 97%|█████████▋| 9701/10000 [1:05:37<01:01,  4.87it/s]
 97%|█████████▋| 9702/10000 [1:05:38<00:59,  4.97it/s]
 97%|█████████▋| 9703/10000 [1:05:38<00:57,  5.18it/s]
 97%|█████████▋| 9704/10000 [1:05:38<00:55,  5.33it/s]
 97%|█████████▋| 9705/10000 [1:05:38<00:55,  5.34it/s]
 97%|█████████▋| 9706/10000 [1:05:38<00:55,  5.32it/s]
 97%|█████████▋| 9707/10000 [1:05:39<00:56,  5.21it/s]
 97%|█████████▋| 9708/10000 [1:05:39<00:56,  5.13it/s]
 97%|█████████▋| 9709/10000 [1:05:39<00:57,  5.02it/s]
 97%|█████████▋| 9710/10000 [1:05:39<00:57,  5.02it/s]
                                                      
{'loss': 0.0792, 'grad_norm': 0.5853787660598755, 'learning_rate': 2.3133606487228397e-07}

 97%|█████████▋| 9710/10000 [1:05:39<00:57,  5.02it/s]
 97%|█████████▋| 9711/10000 [1:05:39<00:56,  5.09it/s]
 97%|█████████▋| 9712/10000 [1:05:40<00:55,  5.23it/s]
 97%|█████████▋| 9713/10000 [1:05:40<00:53,  5.33it/s]
 97%|█████████▋| 9714/10000 [1:05:40<00:52,  5.41it/s]
 97%|█████████▋| 9715/10000 [1:05:40<00:54,  5.26it/s]
 97%|█████████▋| 9716/10000 [1:05:40<00:56,  5.04it/s]
 97%|█████████▋| 9717/10000 [1:05:41<00:58,  4.80it/s]
 97%|█████████▋| 9718/10000 [1:05:41<00:59,  4.72it/s]
 97%|█████████▋| 9719/10000 [1:05:41<01:02,  4.52it/s]
 97%|█████████▋| 9720/10000 [1:05:41<01:03,  4.41it/s]
                                                      
{'loss': 0.0916, 'grad_norm': 0.6415478587150574, 'learning_rate': 2.157211106924295e-07}

 97%|█████████▋| 9720/10000 [1:05:41<01:03,  4.41it/s]
 97%|█████████▋| 9721/10000 [1:05:41<01:00,  4.61it/s]
 97%|█████████▋| 9722/10000 [1:05:42<00:57,  4.85it/s]
 97%|█████████▋| 9723/10000 [1:05:42<00:56,  4.91it/s]
 97%|█████████▋| 9724/10000 [1:05:42<00:57,  4.81it/s]
 97%|█████████▋| 9725/10000 [1:05:42<00:59,  4.60it/s]
 97%|█████████▋| 9726/10000 [1:05:43<01:00,  4.55it/s]
 97%|█████████▋| 9727/10000 [1:05:43<00:59,  4.56it/s]
 97%|█████████▋| 9728/10000 [1:05:43<00:59,  4.61it/s]
 97%|█████████▋| 9729/10000 [1:05:43<00:57,  4.68it/s]
 97%|█████████▋| 9730/10000 [1:05:43<00:56,  4.76it/s]
                                                      
{'loss': 0.0801, 'grad_norm': 0.5180009603500366, 'learning_rate': 2.006505894103672e-07}

 97%|█████████▋| 9730/10000 [1:05:43<00:56,  4.76it/s]
 97%|█████████▋| 9731/10000 [1:05:44<00:55,  4.88it/s]
 97%|█████████▋| 9732/10000 [1:05:44<00:52,  5.09it/s]
 97%|█████████▋| 9733/10000 [1:05:44<00:51,  5.22it/s]
 97%|█████████▋| 9734/10000 [1:05:44<00:49,  5.32it/s]
 97%|█████████▋| 9735/10000 [1:05:44<00:50,  5.21it/s]
 97%|█████████▋| 9736/10000 [1:05:45<00:51,  5.14it/s]
 97%|█████████▋| 9737/10000 [1:05:45<00:55,  4.73it/s]
 97%|█████████▋| 9738/10000 [1:05:45<00:56,  4.60it/s]
 97%|█████████▋| 9739/10000 [1:05:45<00:55,  4.69it/s]
 97%|█████████▋| 9740/10000 [1:05:45<00:53,  4.85it/s]
                                                      
{'loss': 0.067, 'grad_norm': 0.598966121673584, 'learning_rate': 1.8612466583489696e-07}

 97%|█████████▋| 9740/10000 [1:05:45<00:53,  4.85it/s]
 97%|█████████▋| 9741/10000 [1:05:46<00:53,  4.88it/s]
 97%|█████████▋| 9742/10000 [1:05:46<00:50,  5.08it/s]
 97%|█████████▋| 9743/10000 [1:05:46<00:49,  5.21it/s]
 97%|█████████▋| 9744/10000 [1:05:46<00:49,  5.12it/s]
 97%|█████████▋| 9745/10000 [1:05:46<00:51,  4.95it/s]Rank 0, Worker 1: Wait for shard 24 in dataset 0 in 0.00 seconds
Rank 0, Worker 1: Caching shard...

 97%|█████████▋| 9746/10000 [1:05:47<00:54,  4.65it/s]
 97%|█████████▋| 9747/10000 [1:05:47<00:55,  4.56it/s]
 97%|█████████▋| 9748/10000 [1:05:47<00:54,  4.66it/s]
 97%|█████████▋| 9749/10000 [1:05:47<00:52,  4.76it/s]
 98%|█████████▊| 9750/10000 [1:05:47<00:50,  4.94it/s]Rank 0, Worker 0: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 4: Caching shard...Rank 0, Worker 1: Caching shard...

Rank 0, Worker 2: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.69 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.80 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 18.09 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 18.17 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 18.30 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 18.38 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 9750: eval/loss = 0.0807 (36s)

                                                      
{'loss': 0.0753, 'grad_norm': 0.5445523262023926, 'learning_rate': 1.7214349881918834e-07}

 98%|█████████▊| 9750/10000 [1:06:24<00:50,  4.94it/s]
 98%|█████████▊| 9751/10000 [1:06:24<45:58, 11.08s/it]
 98%|█████████▊| 9752/10000 [1:06:24<32:18,  7.82s/it]
 98%|█████████▊| 9753/10000 [1:06:24<22:47,  5.54s/it]
 98%|█████████▊| 9754/10000 [1:06:24<16:08,  3.94s/it]
 98%|█████████▊| 9755/10000 [1:06:25<11:28,  2.81s/it]
 98%|█████████▊| 9756/10000 [1:06:25<08:13,  2.02s/it]
 98%|█████████▊| 9757/10000 [1:06:25<05:57,  1.47s/it]
 98%|█████████▊| 9758/10000 [1:06:25<04:22,  1.09s/it]
 98%|█████████▊| 9759/10000 [1:06:25<03:16,  1.23it/s]
 98%|█████████▊| 9760/10000 [1:06:26<02:29,  1.60it/s]
                                                      
{'loss': 0.065, 'grad_norm': 0.529983401298523, 'learning_rate': 1.5870724125904845e-07}

 98%|█████████▊| 9760/10000 [1:06:26<02:29,  1.60it/s]
 98%|█████████▊| 9761/10000 [1:06:26<01:58,  2.01it/s]
 98%|█████████▊| 9762/10000 [1:06:26<01:36,  2.47it/s]
 98%|█████████▊| 9763/10000 [1:06:26<01:20,  2.94it/s]
 98%|█████████▊| 9764/10000 [1:06:26<01:09,  3.39it/s]
 98%|█████████▊| 9765/10000 [1:06:27<01:01,  3.82it/s]
 98%|█████████▊| 9766/10000 [1:06:27<00:56,  4.15it/s]
 98%|█████████▊| 9767/10000 [1:06:27<00:54,  4.24it/s]
 98%|█████████▊| 9768/10000 [1:06:27<00:51,  4.49it/s]
 98%|█████████▊| 9769/10000 [1:06:27<00:48,  4.77it/s]
 98%|█████████▊| 9770/10000 [1:06:28<00:47,  4.80it/s]
                                                      
{'loss': 0.0553, 'grad_norm': 0.6465544104576111, 'learning_rate': 1.4581604009124006e-07}

 98%|█████████▊| 9770/10000 [1:06:28<00:47,  4.80it/s]
 98%|█████████▊| 9771/10000 [1:06:28<00:46,  4.89it/s]
 98%|█████████▊| 9772/10000 [1:06:28<00:44,  5.09it/s]
 98%|█████████▊| 9773/10000 [1:06:28<00:43,  5.21it/s]
 98%|█████████▊| 9774/10000 [1:06:28<00:42,  5.29it/s]
 98%|█████████▊| 9775/10000 [1:06:28<00:41,  5.37it/s]
 98%|█████████▊| 9776/10000 [1:06:29<00:41,  5.34it/s]
 98%|█████████▊| 9777/10000 [1:06:29<00:41,  5.33it/s]
 98%|█████████▊| 9778/10000 [1:06:29<00:41,  5.34it/s]
 98%|█████████▊| 9779/10000 [1:06:29<00:41,  5.29it/s]
 98%|█████████▊| 9780/10000 [1:06:29<00:41,  5.35it/s]
                                                      
{'loss': 0.06, 'grad_norm': 0.6482387781143188, 'learning_rate': 1.334700362918717e-07}

 98%|█████████▊| 9780/10000 [1:06:29<00:41,  5.35it/s]
 98%|█████████▊| 9781/10000 [1:06:30<00:41,  5.33it/s]
 98%|█████████▊| 9782/10000 [1:06:30<00:40,  5.36it/s]
 98%|█████████▊| 9783/10000 [1:06:30<00:40,  5.41it/s]
 98%|█████████▊| 9784/10000 [1:06:30<00:39,  5.40it/s]
 98%|█████████▊| 9785/10000 [1:06:30<00:40,  5.35it/s]
 98%|█████████▊| 9786/10000 [1:06:31<00:40,  5.30it/s]
 98%|█████████▊| 9787/10000 [1:06:31<00:40,  5.31it/s]
 98%|█████████▊| 9788/10000 [1:06:31<00:39,  5.36it/s]
 98%|█████████▊| 9789/10000 [1:06:31<00:39,  5.40it/s]
 98%|█████████▊| 9790/10000 [1:06:31<00:38,  5.47it/s]
                                                      
{'loss': 0.0663, 'grad_norm': 0.626818060874939, 'learning_rate': 1.2166936487486015e-07}

 98%|█████████▊| 9790/10000 [1:06:31<00:38,  5.47it/s]
 98%|█████████▊| 9791/10000 [1:06:31<00:38,  5.44it/s]
 98%|█████████▊| 9792/10000 [1:06:32<00:37,  5.51it/s]
 98%|█████████▊| 9793/10000 [1:06:32<00:37,  5.56it/s]
 98%|█████████▊| 9794/10000 [1:06:32<00:36,  5.59it/s]
 98%|█████████▊| 9795/10000 [1:06:32<00:36,  5.62it/s]
 98%|█████████▊| 9796/10000 [1:06:32<00:36,  5.62it/s]
 98%|█████████▊| 9797/10000 [1:06:33<00:36,  5.61it/s]
 98%|█████████▊| 9798/10000 [1:06:33<00:36,  5.48it/s]
 98%|█████████▊| 9799/10000 [1:06:33<00:36,  5.45it/s]
 98%|█████████▊| 9800/10000 [1:06:33<00:36,  5.45it/s]
                                                      
{'loss': 0.086, 'grad_norm': 0.6419368386268616, 'learning_rate': 1.1041415489045914e-07}

 98%|█████████▊| 9800/10000 [1:06:33<00:36,  5.45it/s]
 98%|█████████▊| 9801/10000 [1:06:33<00:37,  5.32it/s]
 98%|█████████▊| 9802/10000 [1:06:33<00:36,  5.45it/s]
 98%|█████████▊| 9803/10000 [1:06:34<00:35,  5.52it/s]
 98%|█████████▊| 9804/10000 [1:06:34<00:35,  5.54it/s]
 98%|█████████▊| 9805/10000 [1:06:34<00:35,  5.57it/s]
 98%|█████████▊| 9806/10000 [1:06:34<00:35,  5.48it/s]
 98%|█████████▊| 9807/10000 [1:06:34<00:35,  5.46it/s]
 98%|█████████▊| 9808/10000 [1:06:35<00:34,  5.53it/s]
 98%|█████████▊| 9809/10000 [1:06:35<00:34,  5.58it/s]
 98%|█████████▊| 9810/10000 [1:06:35<00:34,  5.58it/s]
                                                      
{'loss': 0.0636, 'grad_norm': 0.6230601072311401, 'learning_rate': 9.970452942384412e-08}

 98%|█████████▊| 9810/10000 [1:06:35<00:34,  5.58it/s]
 98%|█████████▊| 9811/10000 [1:06:35<00:34,  5.50it/s]
 98%|█████████▊| 9812/10000 [1:06:35<00:33,  5.53it/s]
 98%|█████████▊| 9813/10000 [1:06:35<00:33,  5.59it/s]
 98%|█████████▊| 9814/10000 [1:06:36<00:33,  5.59it/s]
 98%|█████████▊| 9815/10000 [1:06:36<00:32,  5.61it/s]
 98%|█████████▊| 9816/10000 [1:06:36<00:32,  5.58it/s]
 98%|█████████▊| 9817/10000 [1:06:36<00:32,  5.57it/s]
 98%|█████████▊| 9818/10000 [1:06:36<00:32,  5.59it/s]
 98%|█████████▊| 9819/10000 [1:06:36<00:32,  5.56it/s]
 98%|█████████▊| 9820/10000 [1:06:37<00:32,  5.58it/s]
                                                      
{'loss': 0.0549, 'grad_norm': 0.4988340437412262, 'learning_rate': 8.954060559375754e-08}

 98%|█████████▊| 9820/10000 [1:06:37<00:32,  5.58it/s]
 98%|█████████▊| 9821/10000 [1:06:37<00:32,  5.51it/s]
 98%|█████████▊| 9822/10000 [1:06:37<00:32,  5.53it/s]
 98%|█████████▊| 9823/10000 [1:06:37<00:31,  5.54it/s]
 98%|█████████▊| 9824/10000 [1:06:37<00:31,  5.59it/s]
 98%|█████████▊| 9825/10000 [1:06:38<00:31,  5.57it/s]
 98%|█████████▊| 9826/10000 [1:06:38<00:31,  5.58it/s]
 98%|█████████▊| 9827/10000 [1:06:38<00:30,  5.62it/s]
 98%|█████████▊| 9828/10000 [1:06:38<00:30,  5.64it/s]
 98%|█████████▊| 9829/10000 [1:06:38<00:30,  5.55it/s]
 98%|█████████▊| 9830/10000 [1:06:38<00:31,  5.46it/s]
                                                      
{'loss': 0.0737, 'grad_norm': 0.6083027124404907, 'learning_rate': 7.99224945512489e-08}

 98%|█████████▊| 9830/10000 [1:06:39<00:31,  5.46it/s]
 98%|█████████▊| 9831/10000 [1:06:39<00:31,  5.33it/s]
 98%|█████████▊| 9832/10000 [1:06:39<00:31,  5.28it/s]
 98%|█████████▊| 9833/10000 [1:06:39<00:31,  5.37it/s]
 98%|█████████▊| 9834/10000 [1:06:39<00:30,  5.43it/s]
 98%|█████████▊| 9835/10000 [1:06:39<00:30,  5.47it/s]
 98%|█████████▊| 9836/10000 [1:06:40<00:29,  5.51it/s]
 98%|█████████▊| 9837/10000 [1:06:40<00:29,  5.53it/s]
 98%|█████████▊| 9838/10000 [1:06:40<00:29,  5.52it/s]Rank 0, Worker 4: Wait for shard 19 in dataset 0 in 0.00 seconds
Rank 0, Worker 4: Caching shard...

 98%|█████████▊| 9839/10000 [1:06:40<00:29,  5.43it/s]
 98%|█████████▊| 9840/10000 [1:06:40<00:29,  5.48it/s]
                                                      
{'loss': 0.0618, 'grad_norm': 0.5888388156890869, 'learning_rate': 7.085030147843675e-08}

 98%|█████████▊| 9840/10000 [1:06:40<00:29,  5.48it/s]
 98%|█████████▊| 9841/10000 [1:06:41<00:29,  5.40it/s]
 98%|█████████▊| 9842/10000 [1:06:41<00:28,  5.49it/s]Rank 0, Worker 2: Wait for shard 47 in dataset 0 in 0.00 seconds
Rank 0, Worker 2: Caching shard...

 98%|█████████▊| 9843/10000 [1:06:41<00:29,  5.39it/s]
 98%|█████████▊| 9844/10000 [1:06:41<00:29,  5.30it/s]
 98%|█████████▊| 9845/10000 [1:06:41<00:29,  5.21it/s]
 98%|█████████▊| 9846/10000 [1:06:41<00:29,  5.18it/s]
 98%|█████████▊| 9847/10000 [1:06:42<00:29,  5.15it/s]
 98%|█████████▊| 9848/10000 [1:06:42<00:28,  5.29it/s]
 98%|█████████▊| 9849/10000 [1:06:42<00:27,  5.41it/s]
 98%|█████████▊| 9850/10000 [1:06:42<00:28,  5.29it/s]
                                                      
{'loss': 0.0796, 'grad_norm': 0.6196627616882324, 'learning_rate': 6.232412558736523e-08}

 98%|█████████▊| 9850/10000 [1:06:42<00:28,  5.29it/s]
 99%|█████████▊| 9851/10000 [1:06:42<00:28,  5.19it/s]
 99%|█████████▊| 9852/10000 [1:06:43<00:28,  5.15it/s]
 99%|█████████▊| 9853/10000 [1:06:43<00:28,  5.09it/s]
 99%|█████████▊| 9854/10000 [1:06:43<00:28,  5.05it/s]
 99%|█████████▊| 9855/10000 [1:06:43<00:28,  5.07it/s]
 99%|█████████▊| 9856/10000 [1:06:43<00:29,  4.83it/s]
 99%|█████████▊| 9857/10000 [1:06:44<00:29,  4.92it/s]
 99%|█████████▊| 9858/10000 [1:06:44<00:28,  5.01it/s]
 99%|█████████▊| 9859/10000 [1:06:44<00:27,  5.06it/s]
 99%|█████████▊| 9860/10000 [1:06:44<00:27,  5.10it/s]
                                                      
{'loss': 0.0778, 'grad_norm': 0.5279926657676697, 'learning_rate': 5.434406011893822e-08}

 99%|█████████▊| 9860/10000 [1:06:44<00:27,  5.10it/s]
 99%|█████████▊| 9861/10000 [1:06:44<00:27,  4.99it/s]
 99%|█████████▊| 9862/10000 [1:06:45<00:27,  4.97it/s]
 99%|█████████▊| 9863/10000 [1:06:45<00:27,  4.93it/s]
 99%|█████████▊| 9864/10000 [1:06:45<00:27,  4.87it/s]
 99%|█████████▊| 9865/10000 [1:06:45<00:28,  4.73it/s]
 99%|█████████▊| 9866/10000 [1:06:45<00:27,  4.79it/s]
 99%|█████████▊| 9867/10000 [1:06:46<00:27,  4.81it/s]
 99%|█████████▊| 9868/10000 [1:06:46<00:27,  4.79it/s]
 99%|█████████▊| 9869/10000 [1:06:46<00:27,  4.71it/s]
 99%|█████████▊| 9870/10000 [1:06:46<00:27,  4.76it/s]
                                                      
{'loss': 0.0609, 'grad_norm': 0.6097726821899414, 'learning_rate': 4.6910192341864664e-08}

 99%|█████████▊| 9870/10000 [1:06:46<00:27,  4.76it/s]
 99%|█████████▊| 9871/10000 [1:06:47<00:27,  4.71it/s]
 99%|█████████▊| 9872/10000 [1:06:47<00:26,  4.78it/s]
 99%|█████████▊| 9873/10000 [1:06:47<00:26,  4.87it/s]
 99%|█████████▊| 9874/10000 [1:06:47<00:25,  4.89it/s]
 99%|█████████▉| 9875/10000 [1:06:47<00:26,  4.72it/s]
 99%|█████████▉| 9876/10000 [1:06:48<00:27,  4.52it/s]
 99%|█████████▉| 9877/10000 [1:06:48<00:27,  4.39it/s]
 99%|█████████▉| 9878/10000 [1:06:48<00:29,  4.13it/s]
 99%|█████████▉| 9879/10000 [1:06:48<00:29,  4.08it/s]
 99%|█████████▉| 9880/10000 [1:06:49<00:28,  4.16it/s]
                                                      
{'loss': 0.0838, 'grad_norm': 0.5685054063796997, 'learning_rate': 4.0022603551737035e-08}

 99%|█████████▉| 9880/10000 [1:06:49<00:28,  4.16it/s]
 99%|█████████▉| 9881/10000 [1:06:49<00:26,  4.43it/s]
 99%|█████████▉| 9882/10000 [1:06:49<00:25,  4.68it/s]
 99%|█████████▉| 9883/10000 [1:06:49<00:24,  4.81it/s]
 99%|█████████▉| 9884/10000 [1:06:49<00:24,  4.80it/s]
 99%|█████████▉| 9885/10000 [1:06:50<00:24,  4.63it/s]
 99%|█████████▉| 9886/10000 [1:06:50<00:25,  4.42it/s]
 99%|█████████▉| 9887/10000 [1:06:50<00:25,  4.40it/s]
 99%|█████████▉| 9888/10000 [1:06:50<00:25,  4.42it/s]
 99%|█████████▉| 9889/10000 [1:06:51<00:24,  4.49it/s]
 99%|█████████▉| 9890/10000 [1:06:51<00:23,  4.69it/s]
                                                      
{'loss': 0.0573, 'grad_norm': 0.5770753026008606, 'learning_rate': 3.3681369070120985e-08}

 99%|█████████▉| 9890/10000 [1:06:51<00:23,  4.69it/s]
 99%|█████████▉| 9891/10000 [1:06:51<00:22,  4.76it/s]
 99%|█████████▉| 9892/10000 [1:06:51<00:22,  4.82it/s]
 99%|█████████▉| 9893/10000 [1:06:51<00:22,  4.82it/s]
 99%|█████████▉| 9894/10000 [1:06:52<00:21,  4.87it/s]
 99%|█████████▉| 9895/10000 [1:06:52<00:22,  4.72it/s]
 99%|█████████▉| 9896/10000 [1:06:52<00:22,  4.66it/s]
 99%|█████████▉| 9897/10000 [1:06:52<00:22,  4.50it/s]
 99%|█████████▉| 9898/10000 [1:06:52<00:22,  4.49it/s]
 99%|█████████▉| 9899/10000 [1:06:53<00:22,  4.46it/s]
 99%|█████████▉| 9900/10000 [1:06:53<00:21,  4.57it/s]
                                                      
{'loss': 0.0826, 'grad_norm': 0.5914560556411743, 'learning_rate': 2.7886558243744866e-08}

 99%|█████████▉| 9900/10000 [1:06:53<00:21,  4.57it/s]
 99%|█████████▉| 9901/10000 [1:06:53<00:21,  4.60it/s]
 99%|█████████▉| 9902/10000 [1:06:53<00:20,  4.72it/s]
 99%|█████████▉| 9903/10000 [1:06:54<00:20,  4.79it/s]
 99%|█████████▉| 9904/10000 [1:06:54<00:21,  4.56it/s]
 99%|█████████▉| 9905/10000 [1:06:54<00:21,  4.34it/s]
 99%|█████████▉| 9906/10000 [1:06:54<00:21,  4.28it/s]
 99%|█████████▉| 9907/10000 [1:06:54<00:22,  4.21it/s]
 99%|█████████▉| 9908/10000 [1:06:55<00:21,  4.26it/s]
 99%|█████████▉| 9909/10000 [1:06:55<00:20,  4.42it/s]
 99%|█████████▉| 9910/10000 [1:06:55<00:19,  4.53it/s]
                                                      
{'loss': 0.0626, 'grad_norm': 0.6575635671615601, 'learning_rate': 2.2638234443722596e-08}

 99%|█████████▉| 9910/10000 [1:06:55<00:19,  4.53it/s]
 99%|█████████▉| 9911/10000 [1:06:55<00:19,  4.56it/s]
 99%|█████████▉| 9912/10000 [1:06:56<00:18,  4.66it/s]
 99%|█████████▉| 9913/10000 [1:06:56<00:19,  4.57it/s]
 99%|█████████▉| 9914/10000 [1:06:56<00:19,  4.50it/s]
 99%|█████████▉| 9915/10000 [1:06:56<00:20,  4.21it/s]
 99%|█████████▉| 9916/10000 [1:06:57<00:19,  4.24it/s]
 99%|█████████▉| 9917/10000 [1:06:57<00:18,  4.43it/s]
 99%|█████████▉| 9918/10000 [1:06:57<00:17,  4.58it/s]
 99%|█████████▉| 9919/10000 [1:06:57<00:17,  4.76it/s]
 99%|█████████▉| 9920/10000 [1:06:57<00:16,  4.79it/s]
                                                      
{'loss': 0.0657, 'grad_norm': 0.5548856258392334, 'learning_rate': 1.7936455064887504e-08}

 99%|█████████▉| 9920/10000 [1:06:57<00:16,  4.79it/s]
 99%|█████████▉| 9921/10000 [1:06:58<00:16,  4.73it/s]
 99%|█████████▉| 9922/10000 [1:06:58<00:16,  4.77it/s]
 99%|█████████▉| 9923/10000 [1:06:58<00:16,  4.75it/s]
 99%|█████████▉| 9924/10000 [1:06:58<00:15,  4.79it/s]
 99%|█████████▉| 9925/10000 [1:06:58<00:15,  4.86it/s]
 99%|█████████▉| 9926/10000 [1:06:59<00:14,  4.93it/s]
 99%|█████████▉| 9927/10000 [1:06:59<00:14,  5.02it/s]
 99%|█████████▉| 9928/10000 [1:06:59<00:14,  5.04it/s]
 99%|█████████▉| 9929/10000 [1:06:59<00:14,  4.90it/s]
 99%|█████████▉| 9930/10000 [1:06:59<00:14,  4.69it/s]
                                                      
{'loss': 0.0715, 'grad_norm': 0.6243368983268738, 'learning_rate': 1.378127152514841e-08}

 99%|█████████▉| 9930/10000 [1:06:59<00:14,  4.69it/s]
 99%|█████████▉| 9931/10000 [1:07:00<00:14,  4.69it/s]
 99%|█████████▉| 9932/10000 [1:07:00<00:14,  4.59it/s]
 99%|█████████▉| 9933/10000 [1:07:00<00:14,  4.64it/s]
 99%|█████████▉| 9934/10000 [1:07:00<00:13,  4.81it/s]
 99%|█████████▉| 9935/10000 [1:07:00<00:13,  4.94it/s]
 99%|█████████▉| 9936/10000 [1:07:01<00:13,  4.85it/s]
 99%|█████████▉| 9937/10000 [1:07:01<00:13,  4.78it/s]
 99%|█████████▉| 9938/10000 [1:07:01<00:12,  4.79it/s]
 99%|█████████▉| 9939/10000 [1:07:01<00:12,  4.78it/s]
 99%|█████████▉| 9940/10000 [1:07:01<00:12,  4.80it/s]
                                                      
{'loss': 0.0579, 'grad_norm': 0.48955270648002625, 'learning_rate': 1.0172729264917857e-08}

 99%|█████████▉| 9940/10000 [1:07:02<00:12,  4.80it/s]
 99%|█████████▉| 9941/10000 [1:07:02<00:12,  4.83it/s]
 99%|█████████▉| 9942/10000 [1:07:02<00:11,  5.06it/s]
 99%|█████████▉| 9943/10000 [1:07:02<00:10,  5.19it/s]
 99%|█████████▉| 9944/10000 [1:07:02<00:11,  4.87it/s]
 99%|█████████▉| 9945/10000 [1:07:02<00:11,  4.81it/s]
 99%|█████████▉| 9946/10000 [1:07:03<00:11,  4.66it/s]
 99%|█████████▉| 9947/10000 [1:07:03<00:11,  4.65it/s]
 99%|█████████▉| 9948/10000 [1:07:03<00:11,  4.68it/s]
 99%|█████████▉| 9949/10000 [1:07:03<00:11,  4.29it/s]
100%|█████████▉| 9950/10000 [1:07:04<00:11,  4.46it/s]
                                                      
{'loss': 0.0662, 'grad_norm': 0.5606884956359863, 'learning_rate': 7.1108677466458215e-09}

100%|█████████▉| 9950/10000 [1:07:04<00:11,  4.46it/s]
100%|█████████▉| 9951/10000 [1:07:04<00:10,  4.59it/s]
100%|█████████▉| 9952/10000 [1:07:04<00:10,  4.77it/s]
100%|█████████▉| 9953/10000 [1:07:04<00:09,  4.81it/s]
100%|█████████▉| 9954/10000 [1:07:04<00:09,  4.79it/s]
100%|█████████▉| 9955/10000 [1:07:05<00:09,  4.72it/s]
100%|█████████▉| 9956/10000 [1:07:05<00:09,  4.72it/s]
100%|█████████▉| 9957/10000 [1:07:05<00:09,  4.60it/s]
100%|█████████▉| 9958/10000 [1:07:05<00:08,  4.72it/s]
100%|█████████▉| 9959/10000 [1:07:05<00:08,  4.82it/s]
100%|█████████▉| 9960/10000 [1:07:06<00:08,  4.91it/s]
                                                      
{'loss': 0.0667, 'grad_norm': 0.5904876589775085, 'learning_rate': 4.595720454353414e-09}

100%|█████████▉| 9960/10000 [1:07:06<00:08,  4.91it/s]
100%|█████████▉| 9961/10000 [1:07:06<00:07,  4.92it/s]
100%|█████████▉| 9962/10000 [1:07:06<00:07,  5.01it/s]
100%|█████████▉| 9963/10000 [1:07:06<00:07,  4.96it/s]
100%|█████████▉| 9964/10000 [1:07:06<00:07,  4.90it/s]
100%|█████████▉| 9965/10000 [1:07:07<00:07,  4.86it/s]
100%|█████████▉| 9966/10000 [1:07:07<00:07,  4.81it/s]
100%|█████████▉| 9967/10000 [1:07:07<00:06,  4.80it/s]
100%|█████████▉| 9968/10000 [1:07:07<00:06,  4.72it/s]
100%|█████████▉| 9969/10000 [1:07:08<00:06,  4.83it/s]
100%|█████████▉| 9970/10000 [1:07:08<00:06,  4.96it/s]
                                                      
{'loss': 0.0624, 'grad_norm': 0.6997692584991455, 'learning_rate': 2.627314893294264e-09}

100%|█████████▉| 9970/10000 [1:07:08<00:06,  4.96it/s]
100%|█████████▉| 9971/10000 [1:07:08<00:05,  5.02it/s]
100%|█████████▉| 9972/10000 [1:07:08<00:05,  5.10it/s]
100%|█████████▉| 9973/10000 [1:07:08<00:05,  5.01it/s]
100%|█████████▉| 9974/10000 [1:07:09<00:05,  4.87it/s]
100%|█████████▉| 9975/10000 [1:07:09<00:05,  4.73it/s]Rank 0, Worker 3: Wait for shard 2 in dataset 0 in 0.00 seconds
Rank 0, Worker 3: Caching shard...

100%|█████████▉| 9976/10000 [1:07:09<00:05,  4.66it/s]
100%|█████████▉| 9977/10000 [1:07:09<00:04,  4.63it/s]
100%|█████████▉| 9978/10000 [1:07:09<00:04,  4.65it/s]
100%|█████████▉| 9979/10000 [1:07:10<00:04,  4.68it/s]
100%|█████████▉| 9980/10000 [1:07:10<00:04,  4.77it/s]
                                                      
{'loss': 0.0718, 'grad_norm': 0.5520923733711243, 'learning_rate': 1.2056725896270048e-09}

100%|█████████▉| 9980/10000 [1:07:10<00:04,  4.77it/s]
100%|█████████▉| 9981/10000 [1:07:10<00:03,  4.77it/s]
100%|█████████▉| 9982/10000 [1:07:10<00:03,  4.81it/s]
100%|█████████▉| 9983/10000 [1:07:10<00:03,  4.77it/s]
100%|█████████▉| 9984/10000 [1:07:11<00:03,  4.68it/s]
100%|█████████▉| 9985/10000 [1:07:11<00:03,  4.64it/s]
100%|█████████▉| 9986/10000 [1:07:11<00:03,  4.53it/s]
100%|█████████▉| 9987/10000 [1:07:11<00:02,  4.35it/s]
100%|█████████▉| 9988/10000 [1:07:12<00:02,  4.25it/s]
100%|█████████▉| 9989/10000 [1:07:12<00:02,  4.33it/s]
100%|█████████▉| 9990/10000 [1:07:12<00:02,  4.31it/s]
                                                      
{'loss': 0.056, 'grad_norm': 0.6237459182739258, 'learning_rate': 3.308090902098826e-10}

100%|█████████▉| 9990/10000 [1:07:12<00:02,  4.31it/s]
100%|█████████▉| 9991/10000 [1:07:12<00:02,  4.30it/s]
100%|█████████▉| 9992/10000 [1:07:13<00:01,  4.32it/s]
100%|█████████▉| 9993/10000 [1:07:13<00:01,  4.15it/s]
100%|█████████▉| 9994/10000 [1:07:13<00:01,  4.09it/s]
100%|█████████▉| 9995/10000 [1:07:13<00:01,  4.23it/s]
100%|█████████▉| 9996/10000 [1:07:14<00:00,  4.22it/s]
100%|█████████▉| 9997/10000 [1:07:14<00:00,  4.39it/s]
100%|█████████▉| 9998/10000 [1:07:14<00:00,  4.47it/s]
100%|█████████▉| 9999/10000 [1:07:14<00:00,  4.33it/s]
100%|██████████| 10000/10000 [1:07:14<00:00,  4.30it/s]Rank 0, Worker 1: Caching shard...Rank 0, Worker 2: Caching shard...

Rank 0, Worker 0: Caching shard...
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 3: Caching shard...
Rank 0, Worker 5: Wait for shard 5 in dataset 0 in 16.33 seconds
Rank 0, Worker 5: Caching shard...
Rank 0, Worker 4: Wait for shard 1 in dataset 0 in 17.55 seconds
Rank 0, Worker 4: Caching shard...
Rank 0, Worker 1: Wait for shard 4 in dataset 0 in 17.57 seconds
Rank 0, Worker 1: Caching shard...
Rank 0, Worker 0: Wait for shard 7 in dataset 0 in 17.72 seconds
Rank 0, Worker 0: Caching shard...
Rank 0, Worker 2: Wait for shard 2 in dataset 0 in 17.80 seconds
Rank 0, Worker 2: Caching shard...
Rank 0, Worker 3: Wait for shard 3 in dataset 0 in 17.97 seconds
Rank 0, Worker 3: Caching shard...
[held-out-eval] step 10000: eval/loss = 0.0778 (36s)

                                                       
{'loss': 0.0544, 'grad_norm': 0.5916162133216858, 'learning_rate': 2.7339624120159555e-12}

100%|██████████| 10000/10000 [1:07:50<00:00,  4.30it/s]Copying experiment config directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/experiment_cfg to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-10000/experiment_cfg
Copying processor directory /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/processor to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-10000
Copying wandb_config.json from /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/wandb_config.json to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8/checkpoint-10000/wandb_config.json

                                                       
{'train_runtime': 4098.3045, 'train_samples_per_second': 19.52, 'train_steps_per_second': 2.44, 'train_loss': 0.19461959801614284}

100%|██████████| 10000/10000 [1:08:18<00:00,  4.30it/s]
100%|██████████| 10000/10000 [1:08:18<00:00,  2.44it/s]
05/28/2026 11:27:20 - INFO - Model saved to /home/ubuntu/groot-files/checkpoints/run-2026-05-28-101824/groot-wbc-8
05/28/2026 11:27:20 - INFO - Training completed!
wandb: 
wandb: 🚀 View run groot-wbc-8 at: 
wandb: Find logs at: wandb/run-20260528_101832-7coniw77/logs