/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1070: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
[INFO|configuration_utils.py:737] 2024-01-04 22:52:21,055 >> loading configuration file configs/whisper_small_ner.json
[INFO|configuration_utils.py:802] 2024-01-04 22:52:21,057 >> Model config WhisperConfig {
  "_name_or_path": "configs/whisper_small_ner.json",
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "adaptor_layernorm": true,
  "apply_spec_augment": false,
  "architectures": [
    "WhisperForConditionalGeneration"
  ],
  "attention_dropout": 0.0,
  "begin_suppress_tokens": [
    220,
    50257
  ],
  "bos_token_id": 50257,
  "classifier_proj_size": 256,
  "d_model": 768,
  "decoder_attention_heads": 12,
  "decoder_ffn_dim": 3072,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 12,
  "decoder_start_token_id": 50258,
  "dropout": 0.0,
  "encoder_attention_heads": 12,
  "encoder_ffn_dim": 3072,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 12,
  "eos_token_id": 50257,
  "forced_decoder_ids": [
    [
      1,
      50259
    ],
    [
      2,
      50359
    ],
    [
      3,
      50363
    ]
  ],
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "mask_feature_length": 10,
  "mask_feature_min_masks": 0,
  "mask_feature_prob": 0.0,
  "mask_time_length": 10,
  "mask_time_min_masks": 2,
  "mask_time_prob": 0.05,
  "max_length": 448,
  "max_source_positions": 1500,
  "max_target_positions": 448,
  "median_filter_width": 7,
  "model_type": "whisper",
  "num_hidden_layers": 12,
  "num_mel_bins": 80,
  "pad_token_id": 50257,
  "scale_embedding": false,
  "slu_attention_heads": 12,
  "slu_dropout": 0.3,
  "slu_embed_dim": 768,
  "slu_focus": 1.0,
  "slu_input_from": "decoder",
  "slu_input_layers": -1,
  "slu_layers": 2,
  "slu_output_dim": 37,
  "slu_weight": 1.0,
  "suppress_tokens": [
    1,
    2,
    7,
    8,
    9,
    10,
    14,
    25,
    26,
    27,
    28,
    29,
    31,
    58,
    59,
    60,
    61,
    62,
    63,
    90,
    91,
    92,
    93,
    359,
    503,
    522,
    542,
    873,
    893,
    902,
    918,
    922,
    931,
    1350,
    1853,
    1982,
    2460,
    2627,
    3246,
    3253,
    3268,
    3536,
    3846,
    3961,
    4183,
    4667,
    6585,
    6647,
    7273,
    9061,
    9383,
    10428,
    10929,
    11938,
    12033,
    12331,
    12562,
    13793,
    14157,
    14635,
    15265,
    15618,
    16553,
    16604,
    18362,
    18956,
    20075,
    21675,
    22520,
    26130,
    26161,
    26435,
    28279,
    29464,
    31650,
    32302,
    32470,
    36865,
    42863,
    47425,
    49870,
    50254,
    50258,
    50360,
    50361,
    50362
  ],
  "task": "token_classification",
  "torch_dtype": "float32",
  "transformers_version": "4.37.0.dev0",
  "use_cache": true,
  "use_crf": false,
  "use_weighted_layer_sum": false,
  "vocab_size": 51865
}

/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/transformers/models/auto/feature_extraction_auto.py:328: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
[INFO|feature_extraction_utils.py:537] 2024-01-04 22:52:21,311 >> loading configuration file preprocessor_config.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/preprocessor_config.json
[INFO|feature_extraction_utils.py:579] 2024-01-04 22:52:21,317 >> Feature extractor WhisperFeatureExtractor {
  "chunk_length": 30,
  "feature_extractor_type": "WhisperFeatureExtractor",
  "feature_size": 80,
  "hop_length": 160,
  "n_fft": 400,
  "n_samples": 480000,
  "nb_max_frames": 3000,
  "padding_side": "right",
  "padding_value": 0.0,
  "processor_class": "WhisperProcessor",
  "return_attention_mask": false,
  "sampling_rate": 16000
}

/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1899: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
[INFO|tokenization_utils_base.py:2026] 2024-01-04 22:52:21,484 >> loading file vocab.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/vocab.json
[INFO|tokenization_utils_base.py:2026] 2024-01-04 22:52:21,484 >> loading file tokenizer.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/tokenizer.json
[INFO|tokenization_utils_base.py:2026] 2024-01-04 22:52:21,484 >> loading file merges.txt from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/merges.txt
[INFO|tokenization_utils_base.py:2026] 2024-01-04 22:52:21,484 >> loading file normalizer.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/normalizer.json
[INFO|tokenization_utils_base.py:2026] 2024-01-04 22:52:21,484 >> loading file added_tokens.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/added_tokens.json
[INFO|tokenization_utils_base.py:2026] 2024-01-04 22:52:21,484 >> loading file special_tokens_map.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/special_tokens_map.json
[INFO|tokenization_utils_base.py:2026] 2024-01-04 22:52:21,484 >> loading file tokenizer_config.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/tokenizer_config.json
[WARNING|tokenization_utils_base.py:2140] 2024-01-04 22:52:21,484 >> The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'WhisperTokenizer'. 
The class this function is called from is 'NERTokenizerEndToEndFast'.
/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/transformers/modeling_utils.py:2790: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
[INFO|modeling_utils.py:3376] 2024-01-04 22:52:23,048 >> loading weights file model.safetensors from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/model.safetensors
[INFO|configuration_utils.py:826] 2024-01-04 22:52:25,674 >> Generate config GenerationConfig {
  "begin_suppress_tokens": [
    220,
    50257
  ],
  "bos_token_id": 50257,
  "decoder_start_token_id": 50258,
  "eos_token_id": 50257,
  "forced_decoder_ids": [
    [
      1,
      50259
    ],
    [
      2,
      50359
    ],
    [
      3,
      50363
    ]
  ],
  "max_length": 448,
  "pad_token_id": 50257
}

[INFO|modeling_utils.py:4227] 2024-01-04 22:52:26,903 >> All model checkpoint weights were used when initializing WhisperForConditionalGeneration.

[INFO|modeling_utils.py:4235] 2024-01-04 22:52:26,903 >> All the weights of WhisperForConditionalGeneration were initialized from the model checkpoint at openai/whisper-small.
If your task is similar to the task the model of the checkpoint was trained on, you can already use WhisperForConditionalGeneration for predictions without further training.
[INFO|configuration_utils.py:781] 2024-01-04 22:52:27,026 >> loading configuration file generation_config.json from cache at /esat/audioslave/qmeeus/.cache/huggingface/hub/models--openai--whisper-small/snapshots/e34e8ae444c29815eca53e11383ea13b2e362eb0/generation_config.json
[INFO|configuration_utils.py:826] 2024-01-04 22:52:27,027 >> Generate config GenerationConfig {
  "alignment_heads": [
    [
      5,
      3
    ],
    [
      5,
      9
    ],
    [
      8,
      0
    ],
    [
      8,
      4
    ],
    [
      8,
      7
    ],
    [
      8,
      8
    ],
    [
      9,
      0
    ],
    [
      9,
      7
    ],
    [
      9,
      9
    ],
    [
      10,
      5
    ]
  ],
  "begin_suppress_tokens": [
    220,
    50257
  ],
  "bos_token_id": 50257,
  "decoder_start_token_id": 50258,
  "eos_token_id": 50257,
  "forced_decoder_ids": [
    [
      1,
      null
    ],
    [
      2,
      50359
    ]
  ],
  "is_multilingual": true,
  "lang_to_id": {
    "<|af|>": 50327,
    "<|am|>": 50334,
    "<|ar|>": 50272,
    "<|as|>": 50350,
    "<|az|>": 50304,
    "<|ba|>": 50355,
    "<|be|>": 50330,
    "<|bg|>": 50292,
    "<|bn|>": 50302,
    "<|bo|>": 50347,
    "<|br|>": 50309,
    "<|bs|>": 50315,
    "<|ca|>": 50270,
    "<|cs|>": 50283,
    "<|cy|>": 50297,
    "<|da|>": 50285,
    "<|de|>": 50261,
    "<|el|>": 50281,
    "<|en|>": 50259,
    "<|es|>": 50262,
    "<|et|>": 50307,
    "<|eu|>": 50310,
    "<|fa|>": 50300,
    "<|fi|>": 50277,
    "<|fo|>": 50338,
    "<|fr|>": 50265,
    "<|gl|>": 50319,
    "<|gu|>": 50333,
    "<|haw|>": 50352,
    "<|ha|>": 50354,
    "<|he|>": 50279,
    "<|hi|>": 50276,
    "<|hr|>": 50291,
    "<|ht|>": 50339,
    "<|hu|>": 50286,
    "<|hy|>": 50312,
    "<|id|>": 50275,
    "<|is|>": 50311,
    "<|it|>": 50274,
    "<|ja|>": 50266,
    "<|jw|>": 50356,
    "<|ka|>": 50329,
    "<|kk|>": 50316,
    "<|km|>": 50323,
    "<|kn|>": 50306,
    "<|ko|>": 50264,
    "<|la|>": 50294,
    "<|lb|>": 50345,
    "<|ln|>": 50353,
    "<|lo|>": 50336,
    "<|lt|>": 50293,
    "<|lv|>": 50301,
    "<|mg|>": 50349,
    "<|mi|>": 50295,
    "<|mk|>": 50308,
    "<|ml|>": 50296,
    "<|mn|>": 50314,
    "<|mr|>": 50320,
    "<|ms|>": 50282,
    "<|mt|>": 50343,
    "<|my|>": 50346,
    "<|ne|>": 50313,
    "<|nl|>": 50271,
    "<|nn|>": 50342,
    "<|no|>": 50288,
    "<|oc|>": 50328,
    "<|pa|>": 50321,
    "<|pl|>": 50269,
    "<|ps|>": 50340,
    "<|pt|>": 50267,
    "<|ro|>": 50284,
    "<|ru|>": 50263,
    "<|sa|>": 50344,
    "<|sd|>": 50332,
    "<|si|>": 50322,
    "<|sk|>": 50298,
    "<|sl|>": 50305,
    "<|sn|>": 50324,
    "<|so|>": 50326,
    "<|sq|>": 50317,
    "<|sr|>": 50303,
    "<|su|>": 50357,
    "<|sv|>": 50273,
    "<|sw|>": 50318,
    "<|ta|>": 50287,
    "<|te|>": 50299,
    "<|tg|>": 50331,
    "<|th|>": 50289,
    "<|tk|>": 50341,
    "<|tl|>": 50348,
    "<|tr|>": 50268,
    "<|tt|>": 50351,
    "<|uk|>": 50280,
    "<|ur|>": 50290,
    "<|uz|>": 50337,
    "<|vi|>": 50278,
    "<|yi|>": 50335,
    "<|yo|>": 50325,
    "<|zh|>": 50260
  },
  "max_initial_timestamp_index": 1,
  "max_length": 448,
  "no_timestamps_token_id": 50363,
  "pad_token_id": 50257,
  "return_timestamps": false,
  "suppress_tokens": [
    1,
    2,
    7,
    8,
    9,
    10,
    14,
    25,
    26,
    27,
    28,
    29,
    31,
    58,
    59,
    60,
    61,
    62,
    63,
    90,
    91,
    92,
    93,
    359,
    503,
    522,
    542,
    873,
    893,
    902,
    918,
    922,
    931,
    1350,
    1853,
    1982,
    2460,
    2627,
    3246,
    3253,
    3268,
    3536,
    3846,
    3961,
    4183,
    4667,
    6585,
    6647,
    7273,
    9061,
    9383,
    10428,
    10929,
    11938,
    12033,
    12331,
    12562,
    13793,
    14157,
    14635,
    15265,
    15618,
    16553,
    16604,
    18362,
    18956,
    20075,
    21675,
    22520,
    26130,
    26161,
    26435,
    28279,
    29464,
    31650,
    32302,
    32470,
    36865,
    42863,
    47425,
    49870,
    50254,
    50258,
    50358,
    50359,
    50360,
    50361,
    50362
  ],
  "task_to_id": {
    "transcribe": 50359,
    "translate": 50358
  }
}

[INFO|modeling_utils.py:1839] 2024-01-04 22:52:27,073 >> You are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embedding dimension will be 51885. This might induce some performance reduction as *Tensor Cores* will not be available. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc
[INFO|feature_extraction_utils.py:425] 2024-01-04 22:52:42,147 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/preprocessor_config.json
[INFO|tokenization_utils_base.py:2432] 2024-01-04 22:52:42,177 >> tokenizer config file saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tokenizer_config.json
[INFO|tokenization_utils_base.py:2441] 2024-01-04 22:52:42,178 >> Special tokens file saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/special_tokens_map.json
[INFO|configuration_utils.py:483] 2024-01-04 22:52:42,235 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/config.json
[INFO|image_processing_utils.py:373] 2024-01-04 22:52:42,236 >> loading configuration file /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/preprocessor_config.json
[INFO|feature_extraction_utils.py:535] 2024-01-04 22:52:42,236 >> loading configuration file /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/preprocessor_config.json
[INFO|feature_extraction_utils.py:579] 2024-01-04 22:52:42,237 >> Feature extractor WhisperFeatureExtractor {
  "chunk_length": 30,
  "feature_extractor_type": "WhisperFeatureExtractor",
  "feature_size": 80,
  "hop_length": 160,
  "n_fft": 400,
  "n_samples": 480000,
  "nb_max_frames": 3000,
  "padding_side": "right",
  "padding_value": 0.0,
  "processor_class": "WhisperProcessor",
  "return_attention_mask": false,
  "sampling_rate": 16000
}

[INFO|tokenization_utils_base.py:2024] 2024-01-04 22:52:42,239 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2024] 2024-01-04 22:52:42,239 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2024] 2024-01-04 22:52:42,239 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2024] 2024-01-04 22:52:42,239 >> loading file normalizer.json
[INFO|tokenization_utils_base.py:2024] 2024-01-04 22:52:42,239 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2024] 2024-01-04 22:52:42,240 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2024] 2024-01-04 22:52:42,240 >> loading file tokenizer_config.json
[WARNING|tokenization_utils_base.py:2140] 2024-01-04 22:52:42,241 >> The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'NERTokenizerEndToEnd'. 
The class this function is called from is 'WhisperTokenizer'.
[WARNING|logging.py:314] 2024-01-04 22:52:42,323 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|trainer.py:522] 2024-01-04 22:52:42,651 >> max_steps is given, it will override any value given in num_train_epochs
[INFO|trainer.py:571] 2024-01-04 22:52:42,651 >> Using auto half precision backend
[INFO|trainer.py:718] 2024-01-04 22:52:42,771 >> The following columns in the training set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:1712] 2024-01-04 22:52:42,803 >> ***** Running training *****
[INFO|trainer.py:1713] 2024-01-04 22:52:42,803 >>   Num examples = 71,615
[INFO|trainer.py:1714] 2024-01-04 22:52:42,803 >>   Num Epochs = 9
[INFO|trainer.py:1715] 2024-01-04 22:52:42,803 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1718] 2024-01-04 22:52:42,803 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:1719] 2024-01-04 22:52:42,803 >>   Gradient Accumulation steps = 8
[INFO|trainer.py:1720] 2024-01-04 22:52:42,803 >>   Total optimization steps = 5,000
[INFO|trainer.py:1721] 2024-01-04 22:52:42,804 >>   Number of trainable parameters = 153,596,160
[INFO|integration_utils.py:722] 2024-01-04 22:52:42,805 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: Currently logged in as: qmeeus. Use `wandb login --relogin` to force relogin
wandb: wandb version 0.16.1 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.15.12
wandb: Run data is saved locally in /usr/data/condor/execute/dir_27392/whisper_slu/wandb/run-20240104_225244-usvv1760
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run giddy-leaf-131
wandb: ⭐️ View project at https://wandb.ai/qmeeus/WhisperForSpokenNER
wandb: 🚀 View run at https://wandb.ai/qmeeus/WhisperForSpokenNER/runs/usvv1760
[INFO|trainer.py:718] 2024-01-04 23:05:34,670 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-04 23:05:34,672 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-04 23:05:34,672 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-04 23:05:34,672 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-04 23:08:48,979 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-200
[INFO|configuration_utils.py:483] 2024-01-04 23:08:48,982 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-200/config.json
[INFO|configuration_utils.py:594] 2024-01-04 23:08:48,983 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-200/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-04 23:08:51,599 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-200/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-04 23:08:51,601 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-200/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-04 23:19:32,937 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-04 23:19:32,947 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-04 23:19:32,948 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-04 23:19:32,948 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-04 23:22:43,543 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-400
[INFO|configuration_utils.py:483] 2024-01-04 23:22:43,546 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-400/config.json
[INFO|configuration_utils.py:594] 2024-01-04 23:22:43,548 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-400/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-04 23:22:46,776 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-400/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-04 23:22:46,779 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-400/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-04 23:33:36,322 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-04 23:33:36,328 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-04 23:33:36,328 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-04 23:33:36,328 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-04 23:36:46,387 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-600
[INFO|configuration_utils.py:483] 2024-01-04 23:36:46,390 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-600/config.json
[INFO|configuration_utils.py:594] 2024-01-04 23:36:46,392 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-600/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-04 23:36:51,339 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-600/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-04 23:36:51,342 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-600/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-04 23:47:28,826 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-04 23:47:28,828 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-04 23:47:28,828 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-04 23:47:28,828 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-04 23:50:37,963 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-800
[INFO|configuration_utils.py:483] 2024-01-04 23:50:37,965 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-800/config.json
[INFO|configuration_utils.py:594] 2024-01-04 23:50:37,967 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-800/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-04 23:50:40,966 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-800/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-04 23:50:40,968 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-800/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 00:01:16,885 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 00:01:16,886 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 00:01:16,887 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 00:01:16,887 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 00:04:23,392 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1000
[INFO|configuration_utils.py:483] 2024-01-05 00:04:23,395 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1000/config.json
[INFO|configuration_utils.py:594] 2024-01-05 00:04:23,397 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1000/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 00:04:26,837 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1000/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 00:04:26,839 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1000/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 00:15:13,338 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 00:15:13,340 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 00:15:13,340 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 00:15:13,340 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 00:18:21,712 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1200
[INFO|configuration_utils.py:483] 2024-01-05 00:18:21,714 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1200/config.json
[INFO|configuration_utils.py:594] 2024-01-05 00:18:21,716 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1200/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 00:18:29,235 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1200/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 00:18:29,252 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1200/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 00:29:11,761 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 00:29:11,765 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 00:29:11,765 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 00:29:11,765 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 00:32:20,318 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1400
[INFO|configuration_utils.py:483] 2024-01-05 00:32:20,320 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1400/config.json
[INFO|configuration_utils.py:594] 2024-01-05 00:32:20,322 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1400/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 00:32:23,237 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1400/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 00:32:23,239 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1400/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 00:43:02,025 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 00:43:02,027 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 00:43:02,027 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 00:43:02,027 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 00:46:10,035 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1600
[INFO|configuration_utils.py:483] 2024-01-05 00:46:10,038 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1600/config.json
[INFO|configuration_utils.py:594] 2024-01-05 00:46:10,039 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1600/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 00:46:13,625 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1600/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 00:46:13,627 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1600/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 00:56:50,902 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 00:56:50,904 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 00:56:50,904 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 00:56:50,904 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 00:59:59,747 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1800
[INFO|configuration_utils.py:483] 2024-01-05 00:59:59,750 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1800/config.json
[INFO|configuration_utils.py:594] 2024-01-05 00:59:59,752 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1800/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 01:00:03,150 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1800/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 01:00:03,153 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-1800/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 01:10:41,544 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 01:10:41,547 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 01:10:41,547 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 01:10:41,547 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 01:13:49,244 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2000
[INFO|configuration_utils.py:483] 2024-01-05 01:13:49,246 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2000/config.json
[INFO|configuration_utils.py:594] 2024-01-05 01:13:49,247 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2000/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 01:13:52,591 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2000/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 01:13:52,593 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2000/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 01:24:32,011 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 01:24:32,013 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 01:24:32,013 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 01:24:32,013 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 01:27:40,225 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2200
[INFO|configuration_utils.py:483] 2024-01-05 01:27:40,229 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2200/config.json
[INFO|configuration_utils.py:594] 2024-01-05 01:27:40,231 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2200/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 01:27:43,878 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2200/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 01:27:43,880 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2200/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 01:38:21,107 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 01:38:21,111 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 01:38:21,111 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 01:38:21,111 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 01:41:28,684 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2400
[INFO|configuration_utils.py:483] 2024-01-05 01:41:28,686 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2400/config.json
[INFO|configuration_utils.py:594] 2024-01-05 01:41:28,688 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2400/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 01:41:31,566 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2400/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 01:41:31,568 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2400/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 01:52:08,243 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 01:52:08,245 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 01:52:08,245 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 01:52:08,245 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 01:55:16,187 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2600
[INFO|configuration_utils.py:483] 2024-01-05 01:55:16,191 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2600/config.json
[INFO|configuration_utils.py:594] 2024-01-05 01:55:16,193 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2600/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 01:55:19,125 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2600/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 01:55:19,128 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2600/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 02:05:57,426 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 02:05:57,428 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 02:05:57,428 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 02:05:57,428 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 02:09:04,622 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2800
[INFO|configuration_utils.py:483] 2024-01-05 02:09:04,626 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2800/config.json
[INFO|configuration_utils.py:594] 2024-01-05 02:09:04,631 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2800/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 02:09:11,062 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2800/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 02:09:11,064 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-2800/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 02:19:48,993 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 02:19:48,995 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 02:19:48,995 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 02:19:48,995 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 02:22:57,036 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3000
[INFO|configuration_utils.py:483] 2024-01-05 02:22:57,039 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3000/config.json
[INFO|configuration_utils.py:594] 2024-01-05 02:22:57,040 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3000/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 02:22:59,935 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3000/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 02:22:59,938 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3000/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 02:33:36,948 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 02:33:36,950 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 02:33:36,950 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 02:33:36,950 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 02:36:45,028 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3200
[INFO|configuration_utils.py:483] 2024-01-05 02:36:45,030 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3200/config.json
[INFO|configuration_utils.py:594] 2024-01-05 02:36:45,032 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3200/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 02:36:48,007 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3200/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 02:36:48,009 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3200/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 02:47:28,419 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 02:47:28,421 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 02:47:28,421 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 02:47:28,421 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 02:50:36,317 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3400
[INFO|configuration_utils.py:483] 2024-01-05 02:50:36,319 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3400/config.json
[INFO|configuration_utils.py:594] 2024-01-05 02:50:36,321 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3400/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 02:50:44,330 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3400/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 02:50:44,347 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3400/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 03:01:21,635 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 03:01:21,647 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 03:01:21,647 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 03:01:21,647 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 03:04:29,244 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3600
[INFO|configuration_utils.py:483] 2024-01-05 03:04:29,247 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3600/config.json
[INFO|configuration_utils.py:594] 2024-01-05 03:04:29,248 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3600/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 03:04:32,512 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3600/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 03:04:32,514 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3600/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 03:15:11,694 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 03:15:11,695 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 03:15:11,695 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 03:15:11,695 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 03:18:18,870 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3800
[INFO|configuration_utils.py:483] 2024-01-05 03:18:18,872 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3800/config.json
[INFO|configuration_utils.py:594] 2024-01-05 03:18:18,874 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3800/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 03:18:22,360 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3800/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 03:18:22,364 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-3800/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 03:29:00,903 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 03:29:00,905 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 03:29:00,905 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 03:29:00,905 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 03:32:07,479 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4000
[INFO|configuration_utils.py:483] 2024-01-05 03:32:07,481 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4000/config.json
[INFO|configuration_utils.py:594] 2024-01-05 03:32:07,482 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4000/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 03:32:10,449 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4000/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 03:32:10,451 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4000/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 03:42:50,302 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 03:42:50,304 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 03:42:50,304 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 03:42:50,304 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 03:45:57,648 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4200
[INFO|configuration_utils.py:483] 2024-01-05 03:45:57,651 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4200/config.json
[INFO|configuration_utils.py:594] 2024-01-05 03:45:57,652 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4200/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 03:46:00,882 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4200/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 03:46:00,884 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4200/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 03:56:37,348 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 03:56:37,351 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 03:56:37,351 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 03:56:37,351 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 03:59:44,490 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4400
[INFO|configuration_utils.py:483] 2024-01-05 03:59:44,493 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4400/config.json
[INFO|configuration_utils.py:594] 2024-01-05 03:59:44,495 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4400/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 03:59:48,529 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4400/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 03:59:48,531 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4400/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 04:10:27,650 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 04:10:27,653 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 04:10:27,653 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 04:10:27,653 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 04:13:35,434 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4600
[INFO|configuration_utils.py:483] 2024-01-05 04:13:35,437 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4600/config.json
[INFO|configuration_utils.py:594] 2024-01-05 04:13:35,439 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4600/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 04:13:38,099 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4600/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 04:13:38,103 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4600/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 04:24:15,667 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 04:24:15,669 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 04:24:15,669 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 04:24:15,669 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 04:27:23,544 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4800
[INFO|configuration_utils.py:483] 2024-01-05 04:27:23,546 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4800/config.json
[INFO|configuration_utils.py:594] 2024-01-05 04:27:23,548 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4800/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 04:27:26,569 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4800/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 04:27:26,571 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-4800/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 04:38:03,485 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 04:38:03,487 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 04:38:03,487 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 04:38:03,487 >>   Batch size = 8
[INFO|trainer.py:2895] 2024-01-05 04:41:10,446 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-5000
[INFO|configuration_utils.py:483] 2024-01-05 04:41:10,448 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-5000/config.json
[INFO|configuration_utils.py:594] 2024-01-05 04:41:10,449 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-5000/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 04:41:14,677 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-5000/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 04:41:14,679 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/tmp-checkpoint-5000/preprocessor_config.json
[INFO|trainer.py:1953] 2024-01-05 04:41:18,072 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)


[INFO|trainer.py:2895] 2024-01-05 04:41:18,120 >> Saving model checkpoint to /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner
[INFO|configuration_utils.py:483] 2024-01-05 04:41:18,123 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/config.json
[INFO|configuration_utils.py:594] 2024-01-05 04:41:18,125 >> Configuration saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/generation_config.json
[INFO|modeling_utils.py:2413] 2024-01-05 04:41:23,172 >> Model weights saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/model.safetensors
[INFO|feature_extraction_utils.py:425] 2024-01-05 04:41:23,176 >> Feature extractor saved in /esat/audioslave/qmeeus/exp/whisper_slu/e2e/whisper-small-spoken-ner/preprocessor_config.json
[INFO|trainer.py:718] 2024-01-05 04:41:23,183 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
[INFO|trainer.py:3199] 2024-01-05 04:41:23,185 >> ***** Running Evaluation *****
[INFO|trainer.py:3201] 2024-01-05 04:41:23,185 >>   Num examples = 1000
[INFO|trainer.py:3204] 2024-01-05 04:41:23,185 >>   Batch size = 8
wandb: Waiting for W&B process to finish... (success).
wandb: 
wandb: Run history:
wandb:                      eval/loss ▃▂▂▂▁▂▂▁▂▂▂▄▄▄▆▆▇▇▇███████
wandb:                   eval/runtime █▅▄▃▁▃▃▂▃▂▃▂▂▂▂▂▂▂▂▁▂▂▂▂▁▁
wandb:        eval/samples_per_second ▁▄▅▆█▆▆▇▆▇▆▇▇▇▇▇▇▇▇█▇▇▇▇██
wandb:          eval/steps_per_second ▁▄▅▆█▆▆▇▆▇▆▇▇▇▇▇▇▇▇█▇▇▇▇██
wandb:                       eval/wer █▇▆▅▅▄▄▃▃▃▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁
wandb:                    train/epoch ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇████
wandb:              train/global_step ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇████
wandb:            train/learning_rate ▂▄▅▇██████▇▇▇▇▇▆▆▆▆▅▅▅▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁
wandb:                     train/loss █▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:               train/total_flos ▁
wandb:               train/train_loss ▁
wandb:            train/train_runtime ▁
wandb: train/train_samples_per_second ▁
wandb:   train/train_steps_per_second ▁
wandb: 
wandb: Run summary:
wandb:                      eval/loss 0.39327
wandb:                   eval/runtime 186.6982
wandb:        eval/samples_per_second 5.356
wandb:          eval/steps_per_second 0.67
wandb:                       eval/wer 0.14642
wandb:                    train/epoch 8.94
wandb:              train/global_step 5000
wandb:            train/learning_rate 0.0
wandb:                     train/loss 0.0012
wandb:               train/total_flos 1.8469234752159744e+20
wandb:               train/train_loss 0.10567
wandb:            train/train_runtime 20915.2691
wandb: train/train_samples_per_second 30.6
wandb:   train/train_steps_per_second 0.239
wandb: 
wandb: 🚀 View run giddy-leaf-131 at: https://wandb.ai/qmeeus/WhisperForSpokenNER/runs/usvv1760
wandb: ️⚡ View job at https://wandb.ai/qmeeus/WhisperForSpokenNER/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjEyNjkzNDE5MQ==/version_details/v2
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240104_225244-usvv1760/logs
Exception in thread NetStatThr:
Traceback (most recent call last):
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Exception in thread IntMsgThr:
Traceback (most recent call last):
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 267, in check_network_status
    self.run()
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 299, in check_internal_messages
    self._loop_check_status(
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 223, in _loop_check_status
    local_handle = request()
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface.py", line 735, in deliver_network_status
    self._loop_check_status(
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/wandb_run.py", line 223, in _loop_check_status
    local_handle = request()
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface.py", line 743, in deliver_internal_messages
    return self._deliver_network_status(status)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 475, in _deliver_network_status
    return self._deliver_internal_messages(internal_message)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 481, in _deliver_internal_messages
    return self._deliver_record(record)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 428, in _deliver_record
    return self._deliver_record(record)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface_shared.py", line 428, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    handle = mailbox._deliver_record(record, interface=self)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 455, in _deliver_record
    interface._publish(record)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    interface._publish(record)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py", line 51, in _publish
    self._sock_client.send_record_publish(record)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self._sock_client.send_record_publish(record)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 221, in send_record_publish
    self.send_server_request(server_req)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self.send_server_request(server_req)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 155, in send_server_request
    self._send_message(msg)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._send_message(msg)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 152, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    self._sendall_with_error_handle(header + data)
  File "/users/spraak/qmeeus/micromamba/envs/torch-cu121/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe