--- license: cc-by-nc-sa-4.0 metrics: - accuracy - precision - recall - f1 - roc_auc base_model: - facebook/wav2vec2-large-960h-lv60-self --- # 🔍 **AntiDeepfake** The **AntiDeepfake** project provides a series of powerful self-supervised learning (SSL) models crafted to **detect deepfake speech** with state-of-the-art accuracy. # 🤖 Available Models All models are released on Hugging Face 🤗 with two variants: - **Default**: Trained with data augmentation - **NDA** (No Data Augmentation): Trained without data augmentation | Model | Variants | |-----------------------------------------------|--------------------| | XLS-R-2B-AntiDeepfake | [Default](https://huggingface.co/nii-yamagishilab/xls-r-2b-anti-deepfake), [NDA](https://huggingface.co/nii-yamagishilab/xls-r-2b-anti-deepfake-nda) | XLS-R-1B-AntiDeepfake | [Default](https://huggingface.co/nii-yamagishilab/xls-r-1b-anti-deepfake), [NDA](https://huggingface.co/nii-yamagishilab/xls-r-1b-anti-deepfake-nda) | MMS-1B-AntiDeepfake | [Default](https://huggingface.co/nii-yamagishilab/mms-1b-anti-deepfake), [NDA](https://huggingface.co/nii-yamagishilab/mms-1b-anti-deepfake-nda) | MMS-300M-AntiDeepfake | [Default](https://huggingface.co/nii-yamagishilab/mms-300m-anti-deepfake), [NDA](https://huggingface.co/nii-yamagishilab/mms-300m-anti-deepfake-nda) | Wav2Vec2-Large-AntiDeepfake | [Default](https://huggingface.co/nii-yamagishilab/wav2vec-large-anti-deepfake), [NDA](https://huggingface.co/nii-yamagishilab/wav2vec-large-anti-deepfake-nda) | Wav2Vec2-Small-AntiDeepfake | [Default](https://huggingface.co/nii-yamagishilab/wav2vec-small-anti-deepfake), [NDA](https://huggingface.co/nii-yamagishilab/wav2vec-small-anti-deepfake-nda) | Hubert-Extra-Large-AntiDeepfake | [Default](https://huggingface.co/nii-yamagishilab/hubert-xlarge-anti-deepfake), [NDA](https://huggingface.co/nii-yamagishilab/hubert-xlarge-anti-deepfake-nda) # 🛠️ Training Code & Repository Explore training scripts, config files, and evaluation utilities in our GitHub repository:🔗 [AntiDeepfake GitHub Repository](https://github.com/nii-yamagishilab/AntiDeepfake) # 🚀 **Model Spotlight: Wav2Vec2-Large-AntiDeepfake** # 🌟 **Key Features** - **Architecture**: Wav2Vec 2.0 - [`facebook/wav2vec2-large-960h-lv60-self `](https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self/tree/main) 🔗. - **Input**: 16kHz sampled speech with arbitrary length 🎙️. - **Output**: Binary classification score (). - **Training Dataset**: Totally 18k hours of fake speech and 56k hours of real speech. # 🏗️ **Architecture** - **Front-end Feature Extractor**: Wav2Vec2-Large. - **Back-end Classifier**: A fully connected layer. # ⚙️ **Training Details** - **Optimizer**: AdamW with a learning rate of `1e-7`. - **Batch Size**: Dynamic Batching, maximum length per batch is set to 100 seconds. - **Data Augmentation**: RawBoost series: (1)+(2) - **Loss Function**: Cross-Entropy Loss. - **Evaluation Metrics**: Equal Error Rate (EER), ROC AUC, Accuracy, Precision, Recall, F1 Score. # 🚀 **Inference with PyTorch** 📦 Dependencies: ``` ### New conda environments ### conda create --name antideepfake python==3.9.0 conda activate antideepfake conda install pip==24.0 ### Install Fariseq ### # fairseq 0.10.2 on pip does not work git clone https://github.com/pytorch/fairseq cd fairseq # checkout this specific commit. Latest commit does not work git checkout 862efab86f649c04ea31545ce28d13c59560113d pip install --editable . ### Install other packages ### pip install huggingface-hub==0.31.1 safetensors==0.5.3 soundfile==0.13.1 numpy==1.21.2 ``` Additionally, you need to update line 315 in `/where/you/clone/fairseq/fairseq/checkpoint_utils.py` to: ``` state = torch.load(f, map_location=torch.device("cpu"), weights_only=False) ``` 🚀 Inference: ```python import os import urllib.request import torch import torchaudio import fairseq from huggingface_hub import PyTorchModelHubMixin # === Detect all audio files in a specified folder === folder_path = "/path/to/folder/contains/wavs/" audio_formats = (".mp3", ".wav", ".flac", ".m4a") # === Download Fairseq checkpoint if not present === # The downloaded checkpoint is used for building front-end architecture, # its weights will be replaced by the model.safetensors file in this repo. ssl_path = "w2v_large_lv_fsh_swbd_cv.pt" ssl_url = "https://dl.fbaipublicfiles.com/fairseq/wav2vec/w2v_large_lv_fsh_swbd_cv.pt" if not os.path.exists(ssl_path): print(f"Downloading checkpoint to {ssl_path}...") urllib.request.urlretrieve(ssl_url, ssl_path) print("Download complete.") else: print(f"{ssl_path} already exists. Skipping download.") # === Wrapper for the SSL model === class SSLModel(torch.nn.Module): def __init__(self): super().__init__() # The downloaded .pt file is used here model, _, _ = fairseq.checkpoint_utils.load_model_ensemble_and_task([ssl_path]) self.model = model[0] def extract_feat(self, input_data): # If input has shape (B, T, 1), squeeze the last dim if input_data.ndim == 3: input_data = input_data[:, :, 0] # Extract features with torch.no_grad(): features = self.model(input_data, mask=False, features_only=True)['x'] return features # === Function for reading and pre-processing waveforms === def load_wav_and_preprocess(wav_path, target_sr=16000): # Load audio file wav, sr = torchaudio.load(wav_path) # Convert to mono if stereo wav = wav.mean(dim=0) # Resample to target sampling rate wav = torchaudio.functional.resample(wav, sr, new_freq=target_sr) # Normalize waveform with torch.no_grad(): wav = torch.nn.functional.layer_norm(wav, wav.shape) # Add batch dimension and return return wav.unsqueeze(0) # === The actual deepfake detection model using SSL frontend + FC backend === class DeepfakeDetector(torch.nn.Module, PyTorchModelHubMixin): def __init__(self): super().__init__() self.ssl_orig_output_dim = 1024 self.num_classes = 2 # Frontend: SSL model self.m_ssl = SSLModel() # Backend: Pooling + Classification self.adap_pool1d = torch.nn.AdaptiveAvgPool1d(output_size=1) self.proj_fc = torch.nn.Linear( in_features=self.ssl_orig_output_dim, out_features=self.num_classes, ) def forward(self, wav): emb = self.m_ssl.extract_feat(wav) # [B, T, D] emb = emb.transpose(1, 2) # [B, D, T] pooled_emb = self.adap_pool1d(emb) # [B, D, 1] pooled_emb = pooled_emb.squeeze(-1) # [B, D] logits = self.proj_fc(pooled_emb) # [B, 2] return logits # === Load AntiDeepfake model from Hugging Face=== model = DeepfakeDetector.from_pretrained("nii-yamagishilab/wav2vec-large-anti-deepfake") model.eval() # === Inference on a folder of audio files === results = [] for root, _, files in os.walk(folder_path): for file in files: if file.lower().endswith(audio_formats): input_path = os.path.join(root, file) with torch.no_grad(): wav = load_wav_and_preprocess(input_path) logits = model(wav) probs = torch.nn.functional.softmax(logits, dim=1) results.append((file, probs.cpu().numpy()[0])) # Sort results alphabetically by filename results.sort(key=lambda x: x[0]) # Print formatted results print("\n=== Deepfake Detection Results ===") for file_name, prob in results: print(f"{file_name}: real prob = {prob[1]:.3f}, fake prob = {prob[0]:.3f}") ``` # 📊 **Performance Metrics** Results shown below can be reproduced using scripts provided in our [GitHub repository](https://github.com/nii-yamagishilab/AntiDeepfake). | Test Database | ROC AUC | Accuracy | Precision | Recall | F1-score | FPR | FNR | EER (%) @ Threshold | |----------------------|---------|----------|-----------|--------|----------|-------|-------|----------------------| | ADD2023 | 0.950 | 0.912 | 0.939 | 0.942 | 0.940 | 0.175 | 0.058 | 13.25 @ 0.8520 | | DeepVoice | 0.991 | 0.762 | 0.340 | 0.994 | 0.507 | 0.270 | 0.006 | 4.53 @ 0.9974 | | FakeOrReal | 1.000 | 0.992 | 0.994 | 0.989 | 0.991 | 0.005 | 0.011 | 0.63 @ 0.3727 | | FakeOrReal-norm | 0.999 | 0.986 | 0.975 | 0.997 | 0.986 | 0.025 | 0.003 | 0.97 @ 0.7975 | | In-the-Wild | 0.997 | 0.976 | 0.991 | 0.970 | 0.980 | 0.015 | 0.030 | 1.91 @ 0.3240 | | WildSVDD-TEST_A | 0.739 | 0.663 | 0.616 | 0.895 | 0.729 | 0.577 | 0.105 | 34.85 @ 0.9942 | | WildSVDD-TEST_B | 0.654 | 0.696 | 0.740 | 0.836 | 0.785 | 0.583 | 0.164 | 41.67 @ 0.9842 | # **Training Set** Below is a breakdown of the training set used for fine-tuning. | 📚 Database | 🌍 Language | ✅ Genuine (hrs) | ❌ Fake (hrs) | |----------------------|------------------|----------------|-------------| | AISHELL3 | zh | 85.62 | 0 | | ASVspoof2019-LA | en | 11.85 | 97.80 | | ASVspoof2021-LA | en | 16.40 | 116.10 | | ASVspoof2021-DF | en | 20.73 | 487.00 | | ASVspoof5 | en | 413.49 | 1808.48 | | CFAD | zh | 171.25 | 224.55 | | CNCeleb2 | zh | 1084.34 | 0 | | Codecfake | en, zh | 129.66 | 808.32 | | CodecFake | en | 0 | 660.92 | | CVoiceFake | en, fr, de, it, zh| 315.14 | 1561.16 | | DECRO | en, zh | 35.18 | 102.44 | | DFADD | en | 41.62 | 66.01 | | Diffuse or Confuse | en | 0 | 231.66 | | DiffSSD | en | 0 | 139.73 | | DSD | en, ja, ko | 100.98 | 60.23 | | FLEURS | 102 languages | 1388.97 | 0 | | FLEURS-R | 102 languages | 0 | 1238.83 | | HABLA | es | 35.56 | 87.83 | | LibriTTS | en | 585.83 | 0 | | LibriTTS-R | en | 0 | 583.15 | | LibriTTS-Vocoded | en | 0 | 2345.14 | | LJSpeech | en | 23.92 | 0 | | MLADD | 38 languages | 0 | 377.96 | | MLS | 8 languages | 50558.11 | 0 | | SpoofCeleb | Multilingual | 173.00 | 1916.20 | | VoiceMOS | en | 0 | 448.44 | | VoxCeleb2 | Multilingual | 1179.62 | 0 | | VoxCeleb2-Vocoded | Multilingual | 0 | 4721.46 | | WaveFake | en, ja | 0 | 198.65 | | Train Set | Over 100 languages| 56370.00 | 18280.00 | # **Attribution** All AntiDeepfake models were developed by [Yamagishi Lab](https://yamagishilab.jp/) at the National Institute of Informatics (NII), Japan. All model weights are the intellectual property of NII and are made available for research and educational purposes under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license. # **Acknowledgments** This project is based on results obtained from project JPNP22007, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). It is also partially supported by the following grants from the Japan Science and Technology Agency (JST): - AIP Acceleration Research (Grant No. JPMJCR24U3) - CREST (Grant No. JPMJCR20D3) - PRESTO (Grant No. JPMJPR23P9)