Hugging Face's logo

tiny-aya-translate
/
tr-hi-s2st-v0.3

Audio-to-Audio
PEFT
Safetensors
Moshi
Turkish
Hindi
speech-to-speech-translation
simultaneous-translation
mimi
lora
tpu
turkish
hindi
Eval Results (legacy)
Model card Files
xet
 Community
cataluna84 commited on
Commit
d9c1730
·
verified ·
1 Parent(s): 222f1e7

log snapshot step 76000

Browse files
Files changed (1) hide show
  1. logs/train_host0_latest.log +108 -10
logs/train_host0_latest.log CHANGED
@@ -1,14 +1,6 @@
1
- #torch.nn.Module.register_full_backward_hook for more details.
2
  sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
3
- [repeated 12 x]
4
- step 58850 | loss 5.6280 | text 2.0243 audio 5.2231 | grad 1.786 | 1.63s/step | peak -1.0G | host_rss 53.9G
5
- [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=-0.00 G=0.66 | nonfinite=0
6
- sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
7
- [repeated 25 x]
8
- step 58875 | loss 5.6638 | text 2.0330 audio 5.2572 | grad 1.995 | 1.63s/step | peak -1.0G | host_rss 53.9G
9
- [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.01 G=0.76 | nonfinite=0
10
- sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
11
- [repeated 25 x]
12
  step 58900 | loss 5.6608 | text 2.0356 audio 5.2537 | grad 2.533 | 1.70s/step | peak -1.0G | host_rss 53.9G
13
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=0.99 | nonfinite=0
14
  sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
@@ -2840,4 +2832,110 @@ Make sure your token has the correct permissions.
2840
  [ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2841
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2842
  [ckpt] uploading /tmp/ckpt_1zqfzc9x/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_075000
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2843
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
 
1
+ d with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
2
  sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
3
+ [repeated 20 x]
 
 
 
 
 
 
 
 
4
  step 58900 | loss 5.6608 | text 2.0356 audio 5.2537 | grad 2.533 | 1.70s/step | peak -1.0G | host_rss 53.9G
5
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=0.99 | nonfinite=0
6
  sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
 
2832
  [ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2833
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2834
  [ckpt] uploading /tmp/ckpt_1zqfzc9x/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_075000
2835
+ pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
2836
+ [ckpt-index] log failed: Cannot read the W&B step in shared mode.
2837
+ [ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_075000
2838
+ pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
2839
+ step 75025 | loss 5.5113 | text 1.9266 audio 5.1260 | grad 2.047 | 46.33s/step | peak -1.0G | host_rss 36.0G
2840
+ [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=-0.00 G=0.95 | nonfinite=0
2841
+ step 75050 | loss 5.5299 | text 1.9337 audio 5.1432 | grad 1.896 | 1.42s/step | peak -1.0G | host_rss 36.0G
2842
+ [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.09 | spike L=+0.00 G=0.89 | nonfinite=0
2843
+ step 75075 | loss 5.5521 | text 1.9464 audio 5.1628 | grad 1.651 | 1.43s/step | peak -1.0G | host_rss 36.0G
2844
+ [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.01 G=0.78 | nonfinite=0
2845
+ step 75100 | loss 5.5331 | text 1.9260 audio 5.1479 | grad 1.829 | 1.44s/step | peak -1.0G | host_rss 36.0G
2846
+ [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.12 | spike L=+0.00 G=0.88 | nonfinite=0
2847
+ step 75125 | loss 5.5254 | text 1.9286 audio 5.1397 | grad 2.343 | 1.46s/step | peak -1.0G | host_rss 36.0G
2848
+ [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.15 | nonfinite=0
2849
+ step 75150 | loss 5.5237 | text 1.9321 audio 5.1373 | grad 1.384 | 1.43s/step | peak -1.0G | host_rss 36.0G
2850
+ [diag] gradnorm depth=0.55 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.10 | spike L=-0.00 G=0.67 | nonfinite=0
2851
+ step 75175 | loss 5.5475 | text 1.9258 audio 5.1623 | grad 1.638 | 1.43s/step | peak -1.0G | host_rss 36.0G
2852
+ [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.17 | spike L=+0.00 G=0.82 | nonfinite=0
2853
+ step 75200 | loss 5.5273 | text 1.9302 audio 5.1413 | grad 2.348 | 1.44s/step | peak -1.0G | host_rss 36.0G
2854
+ [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.19 | nonfinite=0
2855
+ step 75225 | loss 5.5306 | text 1.9366 audio 5.1433 | grad 2.069 | 1.43s/step | peak -1.0G | host_rss 36.0G
2856
+ [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=1.03 | nonfinite=0
2857
+ step 75250 | loss 5.5234 | text 1.9136 audio 5.1407 | grad 2.088 | 1.44s/step | peak -1.0G | host_rss 36.0G
2858
+ [diag] gradnorm depth=0.38 lora=0.92 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=1.04 | nonfinite=0
2859
+ running validation at step 75250...
2860
+ val/composite=2.8200 (text=0.3978 audio=4.4348) val/loss=4.5144 cb0_acc=41.5% text_acc=96.6%
2861
+ [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
2862
+ [ckpt] uploading /tmp/ckpt_rgf4lnds/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2863
+ [ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2864
+ * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2865
+ step 75275 | loss 5.5107 | text 1.9105 audio 5.1286 | grad 1.871 | 23.85s/step | peak -1.0G | host_rss 36.0G
2866
+ [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.93 | nonfinite=0
2867
+ step 75300 | loss 5.5376 | text 1.9252 audio 5.1526 | grad 1.485 | 1.44s/step | peak -1.0G | host_rss 36.0G
2868
+ [diag] gradnorm depth=0.58 lora=0.80 model_audio_embed=0.05 projection=0.12 text_embed=0.11 | spike L=+0.00 G=0.74 | nonfinite=0
2869
+ step 75325 | loss 5.5086 | text 1.9409 audio 5.1204 | grad 1.995 | 1.44s/step | peak -1.0G | host_rss 36.0G
2870
+ [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.09 | spike L=-0.00 G=1.02 | nonfinite=0
2871
+ step 75350 | loss 5.5494 | text 1.9433 audio 5.1607 | grad 1.708 | 1.44s/step | peak -1.0G | host_rss 36.0G
2872
+ [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.87 | nonfinite=0
2873
+ step 75375 | loss 5.5282 | text 1.9310 audio 5.1420 | grad 1.602 | 1.44s/step | peak -1.0G | host_rss 36.0G
2874
+ [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.83 | nonfinite=0
2875
+ step 75400 | loss 5.5590 | text 1.9338 audio 5.1723 | grad 1.925 | 1.44s/step | peak -1.0G | host_rss 36.0G
2876
+ [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.01 G=1.01 | nonfinite=0
2877
+ step 75425 | loss 5.5113 | text 1.9421 audio 5.1229 | grad 1.688 | 1.44s/step | peak -1.0G | host_rss 36.0G
2878
+ [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.00 G=0.89 | nonfinite=0
2879
+ step 75450 | loss 5.5119 | text 1.9224 audio 5.1274 | grad 1.479 | 1.44s/step | peak -1.0G | host_rss 36.0G
2880
+ [diag] gradnorm depth=0.55 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.00 G=0.79 | nonfinite=0
2881
+ step 75475 | loss 5.5237 | text 1.9185 audio 5.1400 | grad 1.534 | 1.46s/step | peak -1.0G | host_rss 36.0G
2882
+ [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.12 | spike L=-0.00 G=0.83 | nonfinite=0
2883
+ step 75500 | loss 5.5312 | text 1.9159 audio 5.1480 | grad 2.075 | 1.44s/step | peak -1.0G | host_rss 36.0G
2884
+ [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.00 G=1.15 | nonfinite=0
2885
+ running validation at step 75500...
2886
+ val/composite=2.8201 (text=0.3981 audio=4.4348) val/loss=4.5144 cb0_acc=41.5% text_acc=96.6%
2887
+ [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.3% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
2888
+ step 75525 | loss 5.5168 | text 1.9257 audio 5.1317 | grad 2.223 | 2.19s/step | peak -1.0G | host_rss 36.0G
2889
+ [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=-0.00 G=1.21 | nonfinite=0
2890
+ step 75550 | loss 5.5370 | text 1.9294 audio 5.1511 | grad 3.139 | 1.44s/step | peak -1.0G | host_rss 36.0G
2891
+ [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.67 | nonfinite=0
2892
+ step 75575 | loss 5.5511 | text 1.9267 audio 5.1658 | grad 2.855 | 1.44s/step | peak -1.0G | host_rss 36.0G
2893
+ [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.13 | spike L=+0.00 G=1.43 | nonfinite=0
2894
+ step 75600 | loss 5.5343 | text 1.9287 audio 5.1485 | grad 1.969 | 1.43s/step | peak -1.0G | host_rss 36.0G
2895
+ [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.13 | spike L=+0.00 G=0.94 | nonfinite=0
2896
+ step 75625 | loss 5.5495 | text 1.9244 audio 5.1646 | grad 1.777 | 1.46s/step | peak -1.0G | host_rss 36.0G
2897
+ [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.06 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.86 | nonfinite=0
2898
+ step 75650 | loss 5.5488 | text 1.9337 audio 5.1621 | grad 1.607 | 1.46s/step | peak -1.0G | host_rss 36.0G
2899
+ [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.79 | nonfinite=0
2900
+ step 75675 | loss 5.5252 | text 1.9181 audio 5.1416 | grad 1.760 | 1.46s/step | peak -1.0G | host_rss 36.0G
2901
+ [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.88 | nonfinite=0
2902
+ step 75700 | loss 5.5314 | text 1.9281 audio 5.1458 | grad 2.620 | 5.18s/step | peak -1.0G | host_rss 37.0G
2903
+ [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.33 | nonfinite=0
2904
+ step 75725 | loss 5.5297 | text 1.9285 audio 5.1440 | grad 1.616 | 1.47s/step | peak -1.0G | host_rss 37.0G
2905
+ [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.79 | nonfinite=0
2906
+ step 75750 | loss 5.5077 | text 1.9324 audio 5.1212 | grad 1.941 | 1.47s/step | peak -1.0G | host_rss 37.0G
2907
+ [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.97 | nonfinite=0
2908
+ running validation at step 75750...
2909
+ val/composite=2.8199 (text=0.3977 audio=4.4347) val/loss=4.5143 cb0_acc=41.5% text_acc=96.6%
2910
+ [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
2911
+ [ckpt] uploading /tmp/ckpt_4dfalb6j/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2912
+ [ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2913
+ * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2914
+ step 75775 | loss 5.5134 | text 1.9207 audio 5.1293 | grad 2.277 | 23.96s/step | peak -1.0G | host_rss 37.3G
2915
+ [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=-0.00 G=1.14 | nonfinite=0
2916
+ step 75800 | loss 5.5151 | text 1.9138 audio 5.1324 | grad 4.870 | 1.47s/step | peak -1.0G | host_rss 37.3G
2917
+ [diag] gradnorm depth=0.16 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.11 | spike L=-0.00 G=2.41 | nonfinite=0
2918
+ step 75825 | loss 5.5010 | text 1.9115 audio 5.1187 | grad 1.475 | 1.45s/step | peak -1.0G | host_rss 37.3G
2919
+ [diag] gradnorm depth=0.55 lora=0.83 model_audio_embed=0.05 projection=0.11 text_embed=0.09 | spike L=-0.00 G=0.64 | nonfinite=0
2920
+ step 75850 | loss 5.5127 | text 1.9282 audio 5.1271 | grad 1.796 | 1.45s/step | peak -1.0G | host_rss 37.3G
2921
+ [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.81 | nonfinite=0
2922
+ step 75875 | loss 5.5467 | text 1.9228 audio 5.1621 | grad 3.450 | 1.44s/step | peak -1.0G | host_rss 37.2G
2923
+ [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.58 | nonfinite=0
2924
+ step 75900 | loss 5.5455 | text 1.9195 audio 5.1616 | grad 2.258 | 1.44s/step | peak -1.0G | host_rss 37.2G
2925
+ [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.98 | nonfinite=0
2926
+ step 75925 | loss 5.5288 | text 1.9143 audio 5.1460 | grad 1.378 | 4.47s/step | peak -1.0G | host_rss 37.6G
2927
+ [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.04 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.60 | nonfinite=0
2928
+ step 75950 | loss 5.5223 | text 1.9249 audio 5.1373 | grad 2.064 | 1.47s/step | peak -1.0G | host_rss 37.6G
2929
+ [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.93 | nonfinite=0
2930
+ step 75975 | loss 5.5103 | text 1.9413 audio 5.1221 | grad 2.314 | 1.46s/step | peak -1.0G | host_rss 37.6G
2931
+ [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=1.05 | nonfinite=0
2932
+ step 76000 | loss 5.5270 | text 1.9096 audio 5.1451 | grad 2.212 | 1.50s/step | peak -1.0G | host_rss 37.6G
2933
+ [diag] gradnorm depth=0.33 lora=0.94 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=1.00 | nonfinite=0
2934
+ running validation at step 76000...
2935
+ val/composite=2.8199 (text=0.3978 audio=4.4346) val/loss=4.5142 cb0_acc=41.5% text_acc=96.6%
2936
+ [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
2937
+ [ckpt] uploading /tmp/ckpt_47wvo1aj/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2938
+ [ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2939
+ * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
2940
+ [ckpt] uploading /tmp/ckpt_6ga8pi01/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_076000
2941
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3