---
## Executive Summary
As generative AI models (GANs, Diffusion Models, and sophisticated deepfake pipelines like Wav2Lip and Roop) approach total photorealism, human visual inspection is no longer a mathematically reliable metric for media authenticity.
The **Deepfake Forensics Platform** operates as a state-of-the-art digital forensics laboratory. Rather than relying on a monolithic "black-box" classifier, the system implements a **Multi-Modal Ensemble Architecture**. By dissecting media across biological, physical, frequency, and spectral dimensions in real-time, it achieves highly robust detection against out-of-distribution adversarial examples. Furthermore, it integrates **Explainable AI (XAI)** to generate court-grade PDF reports that mathematically justify its verdicts with interpretable visual evidence, heatmaps, and signal plots.
---
## Datasets & Model Training Methodology
This platform relies on a combination of foundational academic weights and custom-trained models tuned specifically for robust deepfake detection.
### 1. Spatial Image Forensics (EfficientNet-B4)
* **Datasets Utilized:** Deepfake Detection Challenge (DFDC), FaceForensics++ (FF++), Celeb-DF, and StyleGAN.
* **Training Methodology:** The core frame-by-frame visual detector utilizes an EfficientNet-B4 backbone. Instead of a simple binary classification approach, the model was fine-tuned using **Contrastive Learning**. By employing a Triplet Loss function, the network was forced to map authentic faces and GAN-generated faces into widely separated clusters in the latent embedding space. It was then capped with a binary cross-entropy classifier. The final convolutional layers (`_conv_head`) are preserved specifically to generate bounding-box localized Grad-CAM heatmaps for XAI tracking.
* **Performance:** Achieved a peak Validation Accuracy of **99.37%** (ROC-AUC 0.998) on a heavily imbalanced dataset of 53,000+ extracted frames.
### 2. Acoustic Anti-Spoofing (Voice Liveness 2D-CNN)
* **Dataset Utilized:** ASVspoof 2019 (Automatic Speaker Verification Spoofing and Countermeasures Challenge) Logical Access (LA) database.
* **Training Methodology:** The `voice_spoofing.pth` model was trained from scratch. The ASVspoof audio tracks were converted into 128-channel Mel-Frequency Spectrograms, effectively treating audio spoofing as an image classification problem. A lightweight PyTorch 2D-CNN was trained to detect the invisible high-frequency spectral rolloffs and vocoder artifacts left behind by TTS engines like ElevenLabs and VITS.
### 3. Native Audio-Visual SyncNet
* **Datasets Utilized:** LRS2 (Lip Reading Sentences 2) and VoxCeleb2.
* **Training Methodology:** This module imports the heavy `syncnet_v2.model` weights originally trained for the Wav2Lip architecture. The model employs a dual-stream 3D-CNN. During training, millions of 5-frame video mouth crops and corresponding 0.2-second audio MFCCs were fed into the network. The network was optimized using contrastive loss to minimize the L2 distance (LSE-D) for synchronized audio-visual pairs, and maximize the distance for artificially shifted, out-of-sync pairs.
### 4. Meta-Classifier Ensemble (XGBoost & Tabular ResNet)
Rather than relying on a single vulnerability, the platform fuses all 15 dimensional anomaly scores into an advanced Tabular ResNet and XGBoost ensemble.
* **Self-Attention Tabular ResNet:** Processes non-sequential forensic metrics (e.g. Spectral Noise vs Geometric Jitter) to learn non-linear correlations between disparate visual and audio anomalies.
* **XAI Meta-Intervention:** If a critical sensor failure is detected (e.g., severe audio spectral rolloff), Explainable AI rules actively intervene to override and boost the final synthetic probability, preventing individual models from drowning out clear deepfake signatures. It was trained using **Soft Labels** (0.15 for Real, 0.85 for Fake) using Binary Cross-Entropy Loss to prevent overconfidence. The synthetic dataset injects advanced probabilistic rules, teaching the Meta-Classifier to flag a video if biological sensors (like rPPG or Geometry) spike, even if the primary Neural Network is successfully fooled by a highly realistic GAN.
---
## Recent Architectural & ML Upgrades
- **Security & Integrity:** Integrated `python-magic` for true binary MIME-type validation to prevent malicious payloads, alongside intelligent **Scene-Cut Detection** via `PySceneDetect` to extract frames across all camera angles.
- **True SHAP Explanations:** The XAI engine utilizes `shap.KernelExplainer` to compute exact marginal contributions from the Meta-Classifier.
- **Batched Inference & Lazy Loading:** Deep learning models are lazy-loaded to conserve idle VRAM, and processing uses chunked batching (sliding window) to prevent GPU OOM errors on large video files.
- **Optimized Face Tracking:** Replaced frame-by-frame Mediapipe face detection with an optimized OpenCV KCF/CSRT tracker, vastly improving pre-processing speed.
- **Real-Time Telemetry:** The FastAPI backend streams progress updates via Server-Sent Events (SSE) instead of traditional HTTP polling.
- **Aesthetic Overhaul (Glassmorphism 2.0):** The React dashboard features a breathtaking "Deep Slate" aesthetic with floating pills, professional typography (`Outfit` and `JetBrains Mono`), and advanced CSS micro-animations.
---
## 15-Dimensional Detection Architecture
The platform executes a massive parallel processing pipeline, routing visual and auditory streams through rigorous forensic methodologies that feed into the final Meta-Classifier Ensemble.
### 1. Neural Network Attention (EfficientNet-B4 + CBAM + XAI)
* **Convolutional Block Attention Module (CBAM):** Integrates custom spatial and channel attention layers into the EfficientNet backbone to aggressively isolate deepfake features.
* **Grad-CAM Heatmaps:** Reverse-engineers the network's spatial attention to generate heatmaps, isolating the exact pixels (e.g., blending boundaries, unnatural eye-reflections) that triggered the synthetic classification.
* **SHAP Feature Importance:** Applies a heuristic-simulated game-theoretic approach to rank which specific forensic dimensions mathematically contributed most to the anomaly variance.
### 2. Spectral & Frequency Analysis
Generative AI inherently struggles to perfectly reconstruct the high-frequency macroscopic details inherent to physical camera sensors.
* **FFT & 2D DCT Spectrum:** Maps two-dimensional frequency coefficients to detect synthetic frequency-domain smoothing.
* **PCA (Principal Component Analysis):** Extracts the 3rd Principal Component (PC3) to reveal hidden periodic GAN artifacts.
* **Switching Noise (SWN):** Isolates high-frequency noise by finding zero-crossings in mathematical gradients, illuminating deepfake splicing seams.
### 3. Biological Face Geometry & Temporal Consistency
Maps 468 3D facial landmarks utilizing **YuNet Face Detection** and **MediaPipe Face Mesh** to evaluate biological impossibility.
* **Temporal Geometric Jitter:** Detects micro-stutters and physically impossible inter-frame vertex shifts, which are common in temporal GAN generation.
* **8-Point Canonical 3D Projection:** Maps the detected face against a rigorous 8-point 3D canonical skull model to robustly compute Head Pose (Pitch, Yaw, Roll) via `cv2.solvePnP` even at extreme angles.
* **Proportional Asymmetry:** Analyzes structural interocular proportions against the facial Golden Ratio using normalized Euclidian distance equations.
### 4. Eye Movement & Dynamic Blink Analysis
* **EAR (Eye Aspect Ratio):** Computes EAR continuously over time to detect unnaturally low blink rates or extreme glitching.
* **Dynamic Median Thresholding:** Unlike hard-coded systems, this pipeline uses dynamic median-based thresholding (80% of resting state) to calculate accurate blink sequences irrespective of diverse human facial structures or camera angles.
* **Gaze Asymmetry:** Detects "lazy eye" artifacts characteristic of poorly rendered generative faces.
### 5. Physical Optics & Sensor Artifacts (CFA & Corneal)
Generative models struggle to accurately simulate physical optics and camera sensor hardware properties.
* **Corneal Specular Highlights:** Maps the reflection of light sources on the eyes. Computes Intersection-over-Union (IoU) and Structural Similarity (SSIM) between the left and right eye reflections. AI models frequently render impossible, mismatched 3D reflections.
* **Color Filter Array (CFA) Artifacts:** Analyzes the Bayer filter interpolation. Genuine digital photos possess distinct periodic demosaicing patterns that AI generators overwrite or fail to produce.
### 6. Native 3D-CNN Audio-Visual Desynchronization (SyncNet)
Armed with the official architecture from **Wav2Lip/SyncNet**, the system catches synthetic "lip-sync" deepfakes by extracting raw audio embeddings and visual lip movements.
* **Deep Embedding L2 Distance:** Extracts 13 MFCC features from the audio and isolated `224x224` visual mouth crops across 5 consecutive frames. Both are passed through independent 3D-CNN encoders.
* **LSE-D & LSE-C:** Mathematically computes the absolute Lip Sync Error Distance (LSE-D). Authentic videos score below `8.0`, while Lip-Sync AI fails to maintain this perfect synchronization, causing the distance to radically diverge.
### 7. Acoustic Anti-Spoofing (Voice Liveness)
Analyzes an audio track for synthetic artifacts common in AI voice clones (e.g. ElevenLabs, VITS) by evaluating Mel-Frequency Spectrograms.
* **Pre-Processing Pipelines:** Utilizes **FFmpeg native scene extraction** for blazingly fast parallel audio/visual splitting. Handles real-world audio corruption via *Cubic Spline De-Clipping* and *Spectral Gating Denoising* prior to inference.
* **Spectral Rolloff & High-Frequency Ratios:** Measures the unnatural high-frequency energy decay often left by generative vocoders.
### 8. Physiological Forensics (rPPG)
Deepfakes frequently fail to synthesize the microscopic, heartbeat-induced color changes in human skin.
* **Remote Photoplethysmography (rPPG):** Extracts subtle volumetric blood flow signals from facial regions of interest using spatial pooling. Applies Fast Fourier Transforms (FFT) to detect if a physiological pulse exists. Generates an anomaly score based on the physiological impossibility of the detected BPM.
### 9. Error Level Analysis (ELA)
Detects heterogeneous compression signatures. When a fake face is spliced onto a real body, the manipulated region possesses a different JPEG compression quality than the original background. Re-saves the image at 95% quality and calculates the absolute pixel-wise difference.
### 10. Temporal Optical Flow & Jitter Analysis
* **DIS (Dense Inverse Search) Optical Flow:** Analyzes temporal consistency on 320x240 resized spatial frames using the blazingly fast DIS algorithm. Computes the variance of motion vectors over a 60-frame buffer to detect micro-jittering, mask boundaries, and blocky temporal flickering common in deepfakes.
### 11. Sensor Noise (PRNU/SRM)
* **Spatial Rich Model (SRM):** Applies high-pass linear filtering to strip away primary image content, isolating the raw noise map. AI-generated face swaps violently disrupt this continuous noise matrix.
### 12. Chrominance Color Space Mapping
Identifies mathematical anomalies in the **YCbCr** (Chrominance separation) and **LAB** (a* channel) spaces, as GANs frequently produce statistical aberrations in human-vision color spaces that are invisible in RGB.
### 13. Cryptographic Metadata Integrity (EXIF)
Analyzes file headers to detect stripped EXIF data or specific cryptographic signatures left behind by generative manipulation software.
---
## Court-Ready PDF Reporting
All automated analyses are compiled into a comprehensive, multi-page PDF report. The document is strictly formatted to provide an interpretable chain-of-evidence:
1. **Executive Verdict:** The overall ensemble confidence score and binary classification.
2. **Detailed Module Breakdown:** Isolated confidence metrics across all analytical engines.
3. **Visual Evidence Gallery:** Embedded high-resolution heatmaps, gradient maps, and XAI overlays.
4. **Metadata Integrity:** Secure UUID assignment and ISO-8601 timestamping.
*(Disclaimer: Reports are generated by automated diagnostic algorithms and should be independently peer-reviewed by a certified forensic analyst prior to legal admission.)*
---
## Getting Started
### Prerequisites
* Python 3.10+
* Node.js (v18+)
* `ffmpeg` installed and globally accessible via the system PATH.
### 1. Initialize the Backend (FastAPI / PyTorch)
The backend is architected for maximum throughput, utilizing a concurrent `ThreadPoolExecutor` to execute heavy OpenCV computations in parallel, bypassing the Python Global Interpreter Lock (GIL).
It is **highly recommended** to use a Virtual Environment to avoid cluttering your global system drive with PyTorch and OpenCV binaries.
```powershell
cd backend
python -m venv venv
.\venv\Scripts\Activate.ps1
pip install -r requirements.txt
uvicorn main:app --reload
```
*The REST API will initialize and bind to `http://127.0.0.1:8000`*
### 2. Initialize the Frontend Dashboard (React / Vite)
The user interface is a responsive, modern React application styled with custom CSS, featuring dark-mode glassmorphism and subtle micro-animations.
```bash
cd frontend
npm install
npm run dev
```
*The analytical dashboard will be accessible at `http://localhost:5173`*
---
## Production Deployment Architecture
This platform is architected for a decoupled, highly-scalable production deployment:
### 1. AI Backend Engine (Hugging Face Spaces)
The FastAPI engine and PyTorch models are designed to be deployed as a Docker container on **Hugging Face Spaces**.
1. Create a new **Docker** Space on Hugging Face.
2. Upload the contents of the `backend/` directory (including `Dockerfile`, `main.py`, `requirements.txt`, etc.) to the root of the Hugging Face Space repository.
3. Ensure the pre-trained weights are uploaded into the `weights/` directory inside the Space.
4. Hugging Face will automatically build the container. The `Dockerfile` is optimized to install CPU-only PyTorch (saving 2.5GB of CUDA libs) and exposes Port `7860`.
5. Once running, your Space will have a URL (e.g., `https://username-spacename.hf.space`).
### 2. Frontend (Vercel)
The React/Vite dashboard is designed to be hosted on **Vercel** for global Edge CDN delivery.
1. Import the `frontend/` directory of this repository into Vercel.
2. In your Vercel Project Settings, add a new Environment Variable:
* Key: `VITE_API_URL`
* Value: The Hugging Face Space URL from the step above (e.g., `https://username-spacename.hf.space`).
3. Deploy! The Vercel instance serves only static assets and handles no heavy computations, deferring all analysis to the HF Space.
---
## Configuration & Constraints
Before deploying the platform, be aware of the following system constraints and configurations:
* **File Upload Limits:** For memory protection during tensor allocations, the API enforces a strict **100 MB** upload limit. Video analysis is capped at the first **60 seconds** of playback. Supported extensions include `mp4`, `avi`, `mov`, `mkv`, `webm`, `png`, `jpg`, and `jpeg`.
* **API Security:** The FastAPI backend is secured via an API Key. By default, it expects the `x-api-key` header to equal `deepforensics-dev-key`. You can override this by setting the `API_KEY` environment variable in the backend, and configuring a `.env` file in the frontend with `VITE_API_KEY=your-key`.
* **Required Model Weights:** Ensure the following pre-trained models are downloaded into the `backend/weights/` directory:
* `improved_finetuned_model.pth` (EfficientNet Backbone)
* `ensemble_mlp.pth` (Meta-Classifier)
* `voice_spoofing.pth` (Audio Anti-Spoofing CNN)
* `syncnet_v2.model` (Wav2Lip Audio-Visual Sync)
---
## System Architecture
```mermaid
flowchart TD
%% Styling Definitions
classDef frontend fill:#3b82f6,stroke:#2563eb,stroke-width:2px,color:#fff,rx:8px,ry:8px;
classDef backend fill:#8b5cf6,stroke:#7c3aed,stroke-width:2px,color:#fff,rx:8px,ry:8px;
classDef processor fill:#10b981,stroke:#059669,stroke-width:2px,color:#fff,rx:8px,ry:8px;
classDef module fill:#1e293b,stroke:#475569,stroke-width:1px,color:#f8fafc,rx:4px,ry:4px;
classDef meta fill:#ef4444,stroke:#dc2626,stroke-width:2px,color:#fff,rx:8px,ry:8px;
classDef output fill:#f59e0b,stroke:#d97706,stroke-width:2px,color:#fff,rx:8px,ry:8px;
%% Client & API Layer
UI[React/Vite Dashboard & Live Terminal]:::frontend -->|Multipart Media Upload| API[FastAPI Gateway w/ MIME Security]:::backend
API -.->|Server-Sent Events: Real-Time Telemetry| UI
API --> VP[Video Processor: OpenCV Tracker & Scene-Cut Split]:::processor
VP --> TP[Concurrent Thread Pool Executor]:::processor
%% The 15-Dimensional Forensic Engines
subgraph Core_Neural_Analysis["Core Neural Analysis"]
NN[EfficientNet-B4 + GradCAM XAI]:::module
end
subgraph Biological_Physiological["Biological & Physiological"]
GEO[Face Geometry & Asymmetry]:::module
EYE[Dynamic Blink & Gaze Analysis]:::module
PHYS[rPPG Volumetric Heartbeat]:::module
end
subgraph Digital_Physical_Optics["Physical Optics & Sensors"]
NOISE[Sensor Noise: PRNU & SRM]:::module
CFA[Bayer CFA Interpolation]:::module
CORNEAL[Corneal Specular Highlights]:::module
LIGHT[Lighting Consistency]:::module
COLOR[Chrominance YCbCr/LAB Mapping]:::module
end
subgraph Temporal_Artifacts["Temporal & Compression"]
ELA[Error Level Analysis]:::module
FLOW[Dense Optical Flow & Jitter]:::module
end
subgraph Audio_Forensics["Acoustic Forensics"]
SYNC[Native 3D-CNN A/V SyncNet]:::module
VOICE[Voice Liveness Anti-Spoofing]:::module
end
subgraph Spectral_Integrity["Spectral & Integrity"]
FA[Frequency: 2D-DCT & FFT]:::module
META[Cryptographic Metadata & EXIF]:::module
end
%% Routing to modules
TP --> NN & GEO & EYE & PHYS & NOISE & CFA & CORNEAL & LIGHT & COLOR & ELA & FLOW & SYNC & VOICE & FA & META
%% Meta-Classifier Aggregation
NN & GEO & EYE & PHYS & NOISE & CFA & CORNEAL & LIGHT & COLOR & ELA & FLOW & SYNC & VOICE & FA & META --> AGG
AGG{Meta-Classifier Ensemble MLP}:::meta
%% Outputs
AGG -->|Inference Complete| PDF[Court-Ready PDF Report Generator]:::output
AGG -->|JSON Response| JSON[REST API JSON Payload]:::output
JSON -->|State Update| UI
PDF -->|Download| UI
```
---
## Codebase Architecture (File Map)
The following diagram maps the high-level logical architecture directly to the underlying physical files and Python/React modules powering the platform:
```mermaid
flowchart TD
subgraph group_frontend["Frontend (React)"]
node_main["Entry point
