README.md

metadata

language: ru
license: mit
library_name: transformers
pipeline_tag: audio-classification
base_model: jonatasgrosman/exp_w2v2t_ru_wavlm_s363
datasets:
  - Aniemore/resd
  - Aniemore/resd_annotated
tags:
  - audio-classification
  - emotion-recognition
  - speech-emotion-recognition
  - audio
  - speech
  - russian
metrics:
  - f1
  - accuracy
  - recall
model-index:
  - name: wavlm-bert-base-s-emotion-russian-resd
    results:
      - task:
          name: Speech Emotion Recognition
          type: audio-classification
        dataset:
          name: RESD test
          type: Aniemore/resd
          args: ru
        metrics:
          - name: Macro F1
            type: f1
            value: 0.7368
          - name: Unweighted accuracy
            type: recall
            value: 0.7406
          - name: Accuracy
            type: accuracy
            value: 0.7429
      - task:
          name: Speech Emotion Recognition
          type: audio-classification
        dataset:
          name: Dusha podcast test
          type: internal
          args: ru
        metrics:
          - name: Macro F1
            type: f1
            value: 0.0927
          - name: Unweighted accuracy
            type: recall
            value: 0.2848
          - name: Accuracy
            type: accuracy
            value: 0.073
      - task:
          name: Speech Emotion Recognition
          type: audio-classification
        dataset:
          name: CAMEO test
          type: internal
          args: ru
        metrics:
          - name: Macro F1
            type: f1
            value: 0.209
          - name: Unweighted accuracy
            type: recall
            value: 0.2354
          - name: Accuracy
            type: accuracy
            value: 0.2566

wavlm-bert-base-s-emotion-russian-resd

Speech emotion recognition for Russian over seven classes: anger, disgust, enthusiasm, fear, happiness, neutral, sadness.

Audio tower jonatasgrosman/exp_w2v2t_ru_wavlm_s363 (WavLM/wav2vec2 fine-tuned for Russian ASR) paired with the text encoder DeepPavlov/rubert-base-cased; the model reads the waveform and a transcript together. Trained on Aniemore/resd.

• Parameters: 493M · weights: 1887 MiB (fp32)
• Input: 16 kHz mono waveform and the utterance transcript
• Quantized builds: wavlm-bert-base-s-emotion-russian-resd-quantized — INT8, FP8 and INT4, up to 5.8x smaller on disk at the same score

Results

Test set	n	UA	WA	macro-F1
RESD test	280	0.7406	0.7429	0.7368
Dusha podcast test	12079	0.2848	0.0730	0.0927
CAMEO test	5187	0.2354	0.2566	0.2090
RESD test, transcript blanked	280	0.7406	0.7429	0.7368

The last row is the audio branch alone. Production rarely has a gold transcript, and the gap between 0.737 with one and 0.737 without is what a transcript is worth here.

Usage

import torch, librosa
from transformers import AutoConfig, AutoTokenizer, AutoModel

repo = "Aniemore/wavlm-bert-base-s-emotion-russian-resd"
cfg = AutoConfig.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained(repo)

# Resample to 16 kHz. RESD ships at 44.1 kHz, and 44.1 kHz audio
# labelled as 16 kHz is stretched 2.8x in time — the model answers,
# it just answers about other audio.
wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
wav = torch.tensor((wav - wav.mean()) / (wav.std() + 1e-7))[None]
t = tok("расшифровка реплики", return_tensors="pt")
with torch.no_grad():
    logits = model(input_ids=t['input_ids'], input_values=wav,
                   text_attention_mask=t['attention_mask'],
                   audio_attention_mask=torch.ones_like(wav).long()).logits

probs = logits.softmax(-1)[0]
print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)})

Loading the audio without librosa

# torchaudio
import torchaudio
wav, sr = torchaudio.load("clip.wav")
wav = torchaudio.functional.resample(wav, sr, 16000).mean(0).numpy()

# torchcodec, the newer decoder
from torchcodec.decoders import AudioDecoder
wav = AudioDecoder("clip.wav", sample_rate=16000).get_all_samples().data.mean(0).numpy()

# straight from the dataset — `datasets` resamples on the column, so
# the mixed 16/44.1 kHz in RESD is handled for you
from datasets import load_dataset, Audio
ds = load_dataset("Aniemore/resd", split="test")
ds = ds.cast_column("speech", Audio(sampling_rate=16000))
wav = ds[0]["speech"]["array"]

Evaluation protocol

Audio resampled to 16 kHz mono, clips capped at 12 s, normalized per utterance, padding masked. UA is macro-averaged recall, WA is accuracy, F1 is macro-averaged. All three test sets went through the same harness, so the rows are comparable to each other.

The RESD split matches fold 1 of EmoBox bit for bit. The top entry there is WavLM-large at WA 56.47 / UA 55.87 / F1 55.82. These numbers are higher, but the training protocol differs — EmoBox freezes the encoder and trains a probe, this is a full fine-tune — so read it as a different recipe on the same split, not as a like-for-like win.

Limitations

RESD is acted and class-balanced. Real speech is neither. The Dusha podcast row above is the honest signal for spontaneous audio, and it is far below the RESD row. Spontaneous Russian is roughly 93% neutral, and a model tuned on balanced acted data over-predicts the emotional classes on it. Measure on your own material, and calibrate the neutral logit if you deploy.

Seven classes only, Russian only, single-speaker clips.

Citation

@misc{aniemore,
  author = {Lubenets, Ilya and Davidchuk, Nikita and Amentes, Aleksandr},
  title  = {Aniemore: an open library for emotion recognition in Russian speech},
  url    = {https://github.com/Aniemore/Aniemore},
  year   = {2023}
}

License

MIT.