---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- Qwen/Qwen3-4B
- Salesforce/blip2-opt-2.7b
tags:
- multimodal
- video-classification
- vision-language
- blip2
- qwen
- content-moderation
- tiktok
- sludge-detection
- lora
---

# Visual-Qwen — TikTok Sludge Detector

A trimodal sludge-content classifier for short-form videos: BLIP-2 (EVA-CLIP-G/14 + Q-Former) + a Linear projector (frozen) + Qwen3-4B with LoRA, augmented with Whisper V3 Turbo audio transcripts. Identifies "sludge" — short videos that stack multiple unrelated visual feeds on one screen, often paired with mismatched audio, engineered to bypass moderation by exploiting unimodal classifiers.

## Performance

Held-out 300-video Kaggle stratified test split (video-level majority vote across 4 sampled frames):

| Metric | Value | 95% CI |
|---|---|---|
| Accuracy | 96.67% | 94.33-98.67 |
| Precision | 95.58% | 92.61-98.31 |
| Recall | 98.86% | 97.06-100.00 |
| F1-score | 97.19% | - |

Confusion matrix (TN=117, FP=8, FN=2, TP=173). 95% CIs from 1,000-iteration bootstrap resampling.

Additional out-of-distribution evaluation on 262 fresh TikToks (scraped May 2026, labelled by Gemma-4-31B-IT): **92.37%** accuracy (95% CI 89.31-95.42). See [the dataset](https://www.kaggle.com/datasets/jobisaacong/tiktok-sludge-dataset-500) `ood-data/` folder for the raw eval set.

## Architecture

```
input frame  ─► EVA-CLIP-G/14 vision encoder (frozen, 1B params)
                          │
                          ▼
              Q-Former + 32 query tokens (frozen)
                          │
                          ▼
              Linear projector (frozen, 4M params)  ◄── audio transcript (Whisper V3 Turbo)
                          │                               │
                          └───────────┬───────────────────┘
                                      ▼
                  Qwen3-4B (4B params) + LoRA r=16 (~32M trainable)
                                      │
                                      ▼
                         {sludge | non-sludge} + explanation
```

The Linear projector is trained in a stage-1 image-caption pre-training pass and then frozen during LoRA fine-tuning on the sludge corpus. Freezing the projector outperformed training it by +0.77 pp on the same eval — see paper section 4.

## Usage

```python
import torch
from transformers import Blip2ForConditionalGeneration, Blip2Processor

MODEL_ID = "alpharomercoma/vqwen-qformer-tiktok-v2"
model = Blip2ForConditionalGeneration.from_pretrained(
    MODEL_ID, dtype=torch.bfloat16, device_map="auto",
)
processor = Blip2Processor.from_pretrained(MODEL_ID)

# Single frame + ASR transcript -> sludge / non-sludge
from PIL import Image
frame = Image.open("frame.jpg").convert("RGB")
transcript = "subway surfers gameplay over family guy clip"  # from Whisper V3 Turbo

messages = [{"role": "user", "content": [
    {"type": "text", "text": f"Audio transcript: {transcript}"},
    {"type": "image"},
    {"type": "text", "text": "Is this sludge content? Answer yes or no."},
]}]
prompt = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=frame, return_tensors="pt").to("cuda")
inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

Run video-level by sampling 4 frames uniformly across the clip and majority-voting the predictions.

## Training data

- 2,000 TikTok and YouTube Shorts videos (1,163 sludge / 837 non-sludge after correction).
- Hand-labelled, then label-refined via a Qwen3-VL-30B-A3B teacher + Gemma-3-27B-IT cross-judge audit pass (231 video flips, 11.55% noise rate in the original GT).
- Multi-task supervision: 25% classify, 15% layout, 20% describe, 35% coupled (classify+explain), 5% honest-refusal templates.

Dataset available on [Kaggle](https://www.kaggle.com/datasets/jobisaacong/tiktok-sludge-dataset-500) — corrected labels, live OOD eval set, and raw mp4s in one dataset.

## Code & paper

- Code: [github.com/alpharomercoma/vqwen-qformer](https://github.com/alpharomercoma/vqwen-qformer)
- Project page: [alpharomercoma.com/thesis](https://alpharomercoma.com/thesis)
- Live demo: [Hugging Face Space](https://huggingface.co/spaces/alpharomercoma/vqwen-qformer-space)

## Citation

```bibtex
@inproceedings{coma2026visualqwen,
  title={Visual-Qwen: Augmenting Multimodal Deep Learning with Attention Mechanisms for Sludge Content Detection},
  author={Coma, Alpha Romer and others},
  year={2026},
  organization={FEU Institute of Technology}
}
```