Spaces:
Running on Zero
Running on Zero
File size: 2,585 Bytes
70af406 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | # Evaluation
This directory contains the evaluation code to reproduce the results from the SAM-Audio paper. The evaluation framework supports multiple datasets, prompting modes (text-only, span, visual), and metrics.
## Setup
Before running evaluation, ensure you have:
1. Installed the SAM-Audio package and its dependencies
2. Authenticated with Hugging Face to access the model checkpoints (see main [README](../README.md))
## Quick Start
Run evaluation on the default setting (instr-pro):
```bash
python main.py
```
You can also use multiple GPUs to speed up evaluation:
```bash
torchrun --nproc_per_node=<ngpus> python main.py
```
Evaluate on a specific setting:
```bash
python main.py --setting sfx
```
Evaluate on multiple settings:
```bash
python main.py --setting sfx speech music
```
## Available Evaluation Settings
Run `python main.py --help` to see all available settings
## Command Line Options
```bash
python main.py [OPTIONS]
```
### Options:
- `-s, --setting` - Which setting(s) to evaluate (default: `instr-pro`)
- Choices: See available settings above
- Can specify multiple settings: `--setting sfx speech music`
- `--cache-path` - Where to cache downloaded datasets (default: `~/.cache/sam_audio`)
- `-p, --checkpoint-path` - Model checkpoint to evaluate (default: `facebook/sam-audio-1b`)
- Can use local path or Hugging Face model ID
- `-b, --batch-size` - Batch size for evaluation (default: `1`)
- `-w, --num-workers` - Number of data loading workers (default: `4`)
- `-c, --candidates` - Number of reranking candidates (default: `8`)
## Evaluation Metrics
The evaluation framework computes the following metrics:
- **Judge** - SAM Audio Judge quality assessment metric
- **Aesthetic** - Aesthetic quality metric
- **CLAP** - Audio-text alignment metric (CLAP similarity)
- **ImageBind** - Audio-video alignment metric (for visual settings only)
## Output
Results are saved to the `results/` directory as JSON files, one per setting:
```
results/
├── sfx.json
├── speech.json
└── music.json
```
Each JSON file contains the averaged metric scores across all samples in that setting.
Example output:
```json
{
"JudgeOverall": "4.386",
"JudgeFaithfulness": "4.708",
"JudgeRecall": "4.934",
"JudgePrecision": "4.451",
"ContentEnjoyment": "5.296",
"ContentUsefulness": "6.903",
"ProductionComplexity": "4.301",
"ProductionQuality": "7.100",
"CLAPSimilarity": "0.271"
}
```
|