Ar4ikov commited on
Commit
d68eb0c
·
verified ·
1 Parent(s): 9c5297d

Add torchaudio, torchcodec and datasets ways to load the audio

Browse files
Files changed (1) hide show
  1. README.md +24 -4
README.md CHANGED
@@ -102,10 +102,9 @@ model = AutoModelForAudioClassification.from_pretrained(
102
  repo, subfolder="int8").eval() # or "fp8", "int4"
103
  fe = AutoFeatureExtractor.from_pretrained(repo, subfolder="int8")
104
 
105
- # librosa resamples and downmixes. Do not skip it: RESD itself ships at
106
- # 44.1 kHz, and handing the model 44.1 kHz audio while telling the
107
- # extractor it is 16 kHz stretches time 2.8x and silently changes the
108
- # answer rather than raising.
109
  wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
110
  x = fe(wav, sampling_rate=16000, return_tensors="pt", padding=True)
111
  with torch.no_grad():
@@ -113,6 +112,27 @@ with torch.no_grad():
113
  print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)})
114
  ```
115
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  ## What is quantized
117
 
118
  Only `nn.Linear` inside the encoder blocks — that is where 98.6% of the weight mass sits. Left in fp32: the convolutional feature extractor, the positional convolution, the layer norms, the projector and the classifier. The classifier is seven rows wide, so quantizing it would save nothing and put rounding error straight onto the logits.
 
102
  repo, subfolder="int8").eval() # or "fp8", "int4"
103
  fe = AutoFeatureExtractor.from_pretrained(repo, subfolder="int8")
104
 
105
+ # Resample to 16 kHz. Do not skip it: RESD itself ships at 44.1 kHz,
106
+ # and handing the model 44.1 kHz audio while telling the extractor it
107
+ # is 16 kHz stretches time 2.8x and silently changes the answer.
 
108
  wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
109
  x = fe(wav, sampling_rate=16000, return_tensors="pt", padding=True)
110
  with torch.no_grad():
 
112
  print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)})
113
  ```
114
 
115
+ <details><summary>Loading the audio without librosa</summary>
116
+
117
+ ```python
118
+ # torchaudio
119
+ import torchaudio
120
+ wav, sr = torchaudio.load("clip.wav")
121
+ wav = torchaudio.functional.resample(wav, sr, 16000).mean(0).numpy()
122
+
123
+ # torchcodec, the newer decoder
124
+ from torchcodec.decoders import AudioDecoder
125
+ wav = AudioDecoder("clip.wav", sample_rate=16000).get_all_samples().data.mean(0).numpy()
126
+
127
+ # straight from the dataset, which resamples on the column
128
+ from datasets import load_dataset, Audio
129
+ ds = load_dataset("Aniemore/resd", split="test")
130
+ ds = ds.cast_column("speech", Audio(sampling_rate=16000))
131
+ wav = ds[0]["speech"]["array"]
132
+ ```
133
+
134
+ </details>
135
+
136
  ## What is quantized
137
 
138
  Only `nn.Linear` inside the encoder blocks &mdash; that is where 98.6% of the weight mass sits. Left in fp32: the convolutional feature extractor, the positional convolution, the layer norms, the projector and the classifier. The classifier is seven rows wide, so quantizing it would save nothing and put rounding error straight onto the logits.