speech-salamandra-es-en Model Card
speech-salamandra-es-en is a speech-language model built on top of salamandra-7b-instruct. It accepts text and optional speech input and it can perform Automatic Speech Recognition (ASR) and Spoken Question Answering (SQA).
DISCLAIMER: This version of Salamandra can expose intermediate reasoning/transcription text. Use
skip_thinking=Truein the helper method, orprocessor.decode_response(..., skip_thinking=True), to display only the final response.
Table of contents
- Model Details
- Intended Use
- Hardware and Software
- How to use
- Training Data
- Evaluation
- Additional Information
Model Details
Description
speech-salamandra-es-en is a finetuned version of salamandra-7b-instruct coupled with the speech encoder from SeamlessM4T v2. It has been trained to follow general text instructions and to emit an internal thinking sequence followed by the final answer.
Intended Use
The model is intended for both research and commercial use on speech/text instruction-following tasks.
Hardware and Software
Training Framework
The code used to train speech-salamandra-es-en is based on the Transformers library, and will be publicly available soon.
Compute Infrastructure
This model was trained on MareNostrum 5, a pre-exascale EuroHPC supercomputer hosted and operated by Barcelona Supercomputing Center.
Training was conducted on 4 nodes, each with the following specifications:
- 4x Nvidia Hopper GPUs with 64GB HBM2 memory
- 2x Intel Sapphire Rapids 8460Y+ at 2.3Ghz and 32c each (64 cores)
- 4x NDR200 (BW per node 800Gb/s)
- 512 GB of Main memory (DDR5)
How to use
Requirements
To use this model, ensure you have the following Python packages installed:transformers>=4.51.2,torch,torchaudio,numpy,soundfile
Load the model and processor directly:
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
model = AutoModelForCausalLM.from_pretrained(
"BSC-LT/speech-salamandra-es-en",
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="auto",
)
processor = AutoProcessor.from_pretrained(
"BSC-LT/speech-salamandra-es-en",
trust_remote_code=True,
)
The convenience method returns the generated text. Set skip_thinking=True to remove the leading reasoning/transcription span and keep only the final answer:
import soundfile as sf
audio_path = "/path/to/audio/example.wav"
audio, sr = sf.read(audio_path)
question = "Who is angry in the audio?"
response = model.generate_response(
processor,
instruction=question,
audio=[(audio, sr)],
max_new_tokens=512,
skip_thinking=True,
)
print(response)
To inspect the full thinking/transcription sequence, keep skip_thinking=False:
full_response = model.generate_response(
processor,
instruction=question,
audio=[(audio, sr)],
max_new_tokens=512,
skip_thinking=False,
)
print(full_response)
For an advanced usage, the generate method can be directly called:
inputs = processor.prepare_inputs(
instruction=question,
audio=[(audio, sr)],
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
input_length = inputs["input_ids"].shape[-1]
inputs["input_features"] = inputs["input_features"].unsqueeze(0).to(
model.device,
dtype=model.dtype,
)
inputs.pop("audio_attention_mask")
inputs["audio_lengths"] = inputs["audio_lengths"].to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
response = processor.decode_response(outputs, input_length=input_length, skip_thinking=True)
Training Data
Global Summary
| Data Type | Hours | Samples |
|---|---|---|
| ASR | 8289.5h | - |
| SQA | 2377h | - |
| IF | - | 327427 |
Automatic Speech Recognition (ASR)
| Dataset | en | es | Total |
|---|---|---|---|
| Common Voice Corpus 22.0 | 1762h | 494h | 2256h |
| VoxPopuli | 501h | 149h | 650h |
| Multilingual Librispeech | 4466.5h | 917h | 5383.5h |
| Total (hours) | 6729.5h | 1560h | 8289.5h |
Spoken Question Answering (SQA)
[NOTE] We refer to SQuADv2 as its subset of non-answerable questions. While we refer to SQuAD as the "answerable" subset in its first version.
We have expanded the SQuAD v2 dataset to speech synthesizing the context of each sample using Kyutai's Pocket TTS. To reduce the length of the audios, before synthesizing, we summarize the contexts with Gemma4 31B.
| Dataset | en | es | Total |
|---|---|---|---|
| LibriSQA | 727h | - | 727h |
| SQuAD | 576h | 525h | 1101h |
| SQuADv2 | 293h | 256h | 549h |
| Total (hours) | 1596h | 781h | 2377h |
Instruction Following Data
Although the model we are using as backbone is already insructed, we still add IF data during the training phase in order to avoid catastrophic forgetting. We use a subset of the original IF data used in the training of ALIA-40B-Instruct-2605. This subset is composed of:
| Dataset | # of samples |
|---|---|
| WildChat | 153392 |
| Aya Dataset | 15146 |
| Mentor | 12719 |
| Other | 39141 |
| Total | 220398 |
Evaluation
Below are the evaluation ASR results on the Common Voice 22.0 and Voxpopuli test sets, and the SQA results on LibriSQA test set and a subset of the SQuAD dataset that we've expanded to synthetic audios. We use as a baseline Voxtral-Mini to compare our model. We report the following metrics:
Click to show metrics details
WER: jiwer implementation. Applied the basic normalization from whisper_normalizer before computing WER.BERTScore: bert_score implementation. We report F1 measure.
Automatic Speech Recognition
Common Voice 22.0 Test
Common Voice 22.0 Test
WER
| SpeechSalamandra | Voxtral | |
|---|---|---|
| en | 9.18 | 11.21 |
| es | 6.67 | 5.44 |
Spoken Question Answering
SQuAD
SQuAD
BERTScore
| SpeechSalamandra | Voxtral | |
|---|---|---|
| en | 0.65 | 0.68 |
| es | 0.58 | 0.63 |
Alternative SQuAD evaluation
For the SQA tasks, the answers include information that is not strictly required by the question. For example:
- Question: How many authors participated in the paper?
- Answer: The number of authors participating in the paper is 5.
- Target Answer: The paper has been written by 5 scientists.
Even though the meaning of both answers is the same, the words used are different. Actually, that is one of the reasons to use BertScore when evaluating this kind of tasks. One way to make this evaluation more robust is to post-edit the output to make the answer as concise as possible. After post-editing, the answers would be something like this:
- Question: How many authors participated in the paper?
- Answer: 5.
- Target Answer: 5.
This practice increases the metrics for both SpeechSalamandra and Voxtral, and make their comparison more focused on “what the model says” and not on “how the model says it”:
| SpeechSalamandra | Voxtral | |
|---|---|---|
| en | 0.78 | 0.82 |
| es | 0.76 | 0.75 |
Additional information
Authors
All the members of the Speech Salamandra Team are members of the AI Institute at the Barcelona Supercomputing Center (BSC).
- Core Contributors (alphabetical order)
- Team Coordination
- Lead
Contact
For further information, please send an email to bsc-lt@bsc.es.
Copyright
Copyright(c) 2026 by AI Institute, Barcelona Supercomputing Center.
Funding
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Modelos del Lenguaje.
Acknowledgements
The training of the model was possible thanks to the computing time provided by Barcelona Supercomputing Center through MareNostrum 5. We acknowledge EuroHPC Joint Undertaking for awarding us access to MareNostrum5 as BSC, Spain.
Disclaimer
Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence.
The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.
License
Citation
If you find our model useful, we would appreciate if you could cite our work as follows:
@misc{bsclt2026speechsalamandra7b ,
title={speech-salamandra-es-en: a Speech LLM model for SQA in English and Spanish.},
author={Speech Salamandra Team},
organization={Barcelona Supercomputing Center},
url={https://huggingface.co/BSC-LT/speech-salamandra-es-en},
year={2026}
}
- Downloads last month
- 22