vqwen-qformer-tiktok

Note — superseded. This is the vision-only TikTok sludge classifier — the ablation baseline for the Visual-Qwen architecture. For deployment, prefer the tri-modal successor vqwen-qformer-tiktok-v2, which adds Whisper-V3-Turbo transcripts at training and inference and lifts frame-level accuracy from 89.00 % to 99.04 % (raw GT) on the same Kaggle test split. This checkpoint remains published as the vision-only baseline row in the multimodal-advantage analysis.

A MiniGPT-4-style vision-language classifier specialised for short-form "sludge" video detection: EVA-CLIP-G/14 + Q-Former + Linear projector + Qwen3-4B. Loads as a stock Blip2ForConditionalGeneration — no trust_remote_code.

Metric (300-video held-out test)	This model	Successor …-v2
Frame-level accuracy (raw GT)	89.00 %	99.04 %
Frame-level accuracy (cleaned GT)	96.70 %	— (n/a)¹
Frame-level precision	91.09 %	99.19 %
Frame-level recall	95.83 %	99.22 %
Frame-level F1	93.40 %	99.21 %
Video-level accuracy (majority vote)	—	99.00 %

¹ The v2 pipeline does not produce a cross-judge-cleaned benchmark. v2's raw-GT 99.04 % already beats this model's cleaned-label 96.70 %.

---

Architecture

Image (224×224)
    → EVA-CLIP-G/14         (frozen, from Salesforce/blip2-opt-2.7b)   → (B, 257, 1408)
    → Q-Former 12-layer     (frozen, 32 pretrained query tokens)        → (B, 32, 768)
    → Linear 768 → 2560     (trained, ~2 M params)                      → (B, 32, 2560)
    → Qwen3-4B              (frozen stage 1; LoRA r=16 merged stage 2)

The 32 is the Q-Former sequence length (number of learnable query tokens), not a feature dim. The Linear projector maps each token from 768 → 2560 to match Qwen3-4B's input embedding space.

The vision tower is EVA-CLIP-G/14 (BAAI's EVA-01-CLIP-g/14, accessed through the BLIP-2 bundle); hidden_size = 1408. Verify against config.vision_config.hidden_size.

Intended use

Binary classification of whether a short-form video frame belongs to a "sludge" layout — two or more unrelated visual streams shown simultaneously (split-screen, picture-in-picture, or collage), typically paired with one stream's audio.

Also supports (from multi-task training):

• Layout categorisation (vertical, horizontal, pip, other).
• Per-frame visual description grounded in visible content (no fabricated show/game/channel names).
• Coupled classify+explain.
• Safe refusal on "which specific show / game / channel" questions.

This model does not consume audio transcripts. If a transcript is available at inference time, use the v2 successor instead.

Usage

import torch
from PIL import Image
from transformers import Blip2ForConditionalGeneration, AutoProcessor

MODEL_ID = "alpharomercoma/vqwen-qformer-tiktok"
model = Blip2ForConditionalGeneration.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)

image = Image.open("frame.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Is this sludge content? Answer yes or no."},
    ],
}]
prompt = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
inputs["pixel_values"] = inputs["pixel_values"].to(dtype=torch.bfloat16)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=8, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use processor.tokenizer.apply_chat_template, not processor.apply_chat_template — the chat template is persisted on the tokenizer.

Training

• Stage 1 (alignment, frozen): linear projector pretrained on liuhaotian/LLaVA-Pretrain 558 K. Reused from alpharomercoma/vqwen-qformer-pretrain.
• Stage 2 (this checkpoint): LoRA fine-tune via teacher-student distillation. Qwen/Qwen3-VL-30B-A3B-Instruct labelled 8,487 TikTok frames (~1,700 videos × ≤5 frames) with structured JSON {sludge, layout, description, explanation}; google/gemma-3-27b-it cross-judged 250 teacher↔GT disagreements (teacher won 232/250 = 92.8 %, confirming ~10–12 % semantic label noise in raw GT).
• Task mix (one random task per frame): classify 25 % / layout 15 % / describe (teacher) 20 % / coupled-explain 35 % / refuse 5 %.
• LoRA: r=16, α=32, dropout=0.15, targets [q,k,v,o,gate,up,down]_proj; ~35 M trainable.
• Optimisation: bf16, fused AdamW, cosine schedule, warmup 0.03, LR 2e-4 (LoRA) / 2e-5 (projector), global batch 128, 10 epochs (≈530 steps). Eval every 25 steps, load_best_model_at_end on eval_loss.
• Hardware: single NVIDIA H200 141 GB. Runtime: ~12 min train + ~1.5 h teacher labelling pass.

Evaluation — 300-video held-out test split

Benchmark	correct	accuracy
vs raw human GT	267 / 300	89.00 %
vs Gemma-3-cleaned labels	290 / 300	96.70 %

The 7.7-pp gap reflects the ~10–12 % semantic noise in raw human labels under the strict sludge definition (Gemma sided with the teacher on 92.8 % of disagreements). Zero hallucinated show/game/channel mentions in the descriptions.

Limitations

• No audio modality. This is the vision-only ablation; use the v2 successor for the multimodal pipeline.
• English-only output; single-frame reasoning (no temporal context).
• Will not name specific shows/games/creators even when legible in the frame (the training labels deliberately suppress that).
• Trained on short-form vertical TikTok / YT-Shorts content; generalisation to long-form video or other multi-pane formats is untested.

Citation

@misc{vqwenqformertiktok,
  author = {Olata, Marc and Coma, Alpha Romer and Ong, Job Isaac and Sioson, Kristoffer Ian},
  title  = {Visual-Qwen: Augmenting Multimodal Deep Learning with Attention Mechanisms to Recognize "Sludge" Videos from Short-Form Content},
  year   = {2026},
  note   = {vision-only ablation baseline},
  publisher = {FEU Institute of Technology}
}

Underlying components:

• BLIP-2 (Li et al., 2023, arXiv:2301.12597)
• EVA-CLIP (Sun et al., 2023, arXiv:2303.15389)
• Qwen3 (Yang et al., 2025; Qwen Team)
• Teacher: Qwen/Qwen3-VL-30B-A3B-Instruct; Judge: google/gemma-3-27b-it

License

Apache 2.0 for the trained deltas (LoRA + projector). Base models retain their original licenses: Salesforce/blip2-opt-2.7b (BSD-3), Qwen/Qwen3-4B (Apache 2.0).