---
language: ru
license: mit
library_name: transformers
pipeline_tag: audio-classification
base_model: jonatasgrosman/exp_w2v2t_ru_wavlm_s363
datasets:
- Aniemore/resd
- Aniemore/resd_annotated
tags:
- audio-classification
- emotion-recognition
- speech-emotion-recognition
- audio
- speech
- russian
metrics:
- f1
- accuracy
- recall
model-index:
- name: wavlm-bert-base-s-emotion-russian-resd
results:
- task:
name: Speech Emotion Recognition
type: audio-classification
dataset:
name: RESD test
type: Aniemore/resd
args: ru
metrics:
- name: Macro F1
type: f1
value: 0.7368
- name: Unweighted accuracy
type: recall
value: 0.7406
- name: Accuracy
type: accuracy
value: 0.7429
- task:
name: Speech Emotion Recognition
type: audio-classification
dataset:
name: Dusha podcast test
type: internal
args: ru
metrics:
- name: Macro F1
type: f1
value: 0.0927
- name: Unweighted accuracy
type: recall
value: 0.2848
- name: Accuracy
type: accuracy
value: 0.0730
- task:
name: Speech Emotion Recognition
type: audio-classification
dataset:
name: CAMEO test
type: internal
args: ru
metrics:
- name: Macro F1
type: f1
value: 0.2090
- name: Unweighted accuracy
type: recall
value: 0.2354
- name: Accuracy
type: accuracy
value: 0.2566
---
# wavlm-bert-base-s-emotion-russian-resd
Speech emotion recognition for Russian over seven classes: `anger`, `disgust`, `enthusiasm`, `fear`, `happiness`, `neutral`, `sadness`.
Audio tower [`jonatasgrosman/exp_w2v2t_ru_wavlm_s363`](https://huggingface.co/jonatasgrosman/exp_w2v2t_ru_wavlm_s363) (WavLM/wav2vec2 fine-tuned for Russian ASR) paired with the text encoder [`DeepPavlov/rubert-base-cased`](https://huggingface.co/DeepPavlov/rubert-base-cased); the model reads the waveform and a transcript together. Trained on [`Aniemore/resd`](https://huggingface.co/datasets/Aniemore/resd).
- **Parameters**: 493M · **weights**: 1887 MiB (fp32)
- **Input**: 16 kHz mono waveform and the utterance transcript
- **Quantized builds**: [`wavlm-bert-base-s-emotion-russian-resd-quantized`](https://huggingface.co/Aniemore/wavlm-bert-base-s-emotion-russian-resd-quantized) — INT8, FP8 and INT4, up to 5.8x smaller on disk at the same score
## Results
| Test set | n | UA | WA | macro-F1 |
|---|---:|---:|---:|---:|
| RESD test | 280 | 0.7406 | 0.7429 | 0.7368 |
| Dusha podcast test | 12079 | 0.2848 | 0.0730 | 0.0927 |
| CAMEO test | 5187 | 0.2354 | 0.2566 | 0.2090 |
| RESD test, transcript blanked | 280 | 0.7406 | 0.7429 | 0.7368 |
The last row is the audio branch alone. Production rarely has a gold transcript, and the gap between 0.737 with one and 0.737 without is what a transcript is worth here.
## Usage
```python
import torch, librosa
from transformers import AutoConfig, AutoTokenizer, AutoModel
repo = "Aniemore/wavlm-bert-base-s-emotion-russian-resd"
cfg = AutoConfig.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained(repo)
# Resample to 16 kHz. RESD ships at 44.1 kHz, and 44.1 kHz audio
# labelled as 16 kHz is stretched 2.8x in time — the model answers,
# it just answers about other audio.
wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
wav = torch.tensor((wav - wav.mean()) / (wav.std() + 1e-7))[None]
t = tok("расшифровка реплики", return_tensors="pt")
with torch.no_grad():
logits = model(input_ids=t['input_ids'], input_values=wav,
text_attention_mask=t['attention_mask'],
audio_attention_mask=torch.ones_like(wav).long()).logits
probs = logits.softmax(-1)[0]
print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)})
```
Loading the audio without librosa
```python
# torchaudio
import torchaudio
wav, sr = torchaudio.load("clip.wav")
wav = torchaudio.functional.resample(wav, sr, 16000).mean(0).numpy()
# torchcodec, the newer decoder
from torchcodec.decoders import AudioDecoder
wav = AudioDecoder("clip.wav", sample_rate=16000).get_all_samples().data.mean(0).numpy()
# straight from the dataset — `datasets` resamples on the column, so
# the mixed 16/44.1 kHz in RESD is handled for you
from datasets import load_dataset, Audio
ds = load_dataset("Aniemore/resd", split="test")
ds = ds.cast_column("speech", Audio(sampling_rate=16000))
wav = ds[0]["speech"]["array"]
```