PenguinVL
Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
---
## ๐ฐ News
* **2025.03** โ PenguinVL-Encoder now available for general use.
* **2025.03** โ Released PenguinVL-2B, PenguinVL-8B.
---
## ๐ Model Overview
PenguinVL is a compact Vision-Language Model, designed to explore the efficiency limits of small-scale VLMs.
Unlike most existing VLMs that rely on contrastive-pretrained vision encoders (e.g., CLIP/SigLIP), PG-VL initializes its vision encoder directly from a **text-only LLM**. This design avoids the objective mismatch between contrastive learning and autoregressive language modeling, enabling tighter alignment between visual representations and the language backbone.
### Key Characteristics
- ๐ง **LLM-based Vision Encoder**
The vision encoder is adapted from a pretrained text LLM (Qwen3-0.6B), modified with bidirectional attention and 2D-RoPE for spatial modeling.
This provides strong semantic priors and native compatibility with the downstream LLM.
- ๐ฅ **Efficient Video Understanding**
A Temporal Redundancy-Aware (TRA) token compression strategy dynamically allocates token budgets across frames, enabling long-video reasoning within a limited context window.
- ๐ Unified Architecture
The model consists of:
1. LLM-initialized vision encoder
2. Lightweight MLP projector
3. Qwen3 language backbone
- ๐ Compact but Strong
At 2B scale, PG-VL achieves competitive performance across image, document, OCR, math, and video benchmarks while remaining deployment-friendly.
---
## ๐งช Quick Start โ Transformers Inference
```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
model_name = "pg-team/pg-vl-2b-hf"
model = AutoModelForCausalLM.from_pretrained(
model_name,
trust_remote_code=True,
device_map="auto",
torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
# Example: Image + Text
inputs = processor(
conversation=[
{"role": "system", "content": "You are a helpful assistant."},
{
"role": "user",
"content": [
{"type": "image", "image": {"image_path": "assets/example.jpg"}},
{"type": "text", "text": "Describe this image."}
],
},
],
return_tensors="pt",
)
inputs = {k: v.to("cuda") for k, v in inputs.items() if isinstance(v, torch.Tensor)}
output_ids = model.generate(**inputs, max_new_tokens=128)
response = processor.decode(output_ids[0], skip_special_tokens=True)
print(response)
```
## ๐ Model Zoo
| Model | Base Model | HF Link |
| -------------------- | ------------ | ------------------------------------------------------------ |
| PenguinVL-8B | Qwen3-8B | [pg-team/pg-vl-8b-hf](https://huggingface.co/pg-team/pg-vl-8b-hf) |
| PenguinVL-2B | Qwen3-1.7B | [pg-team/pg-vl-2b-hf](https://huggingface.co/pg-team/pg-vl-2b-hf) |
| PenguinVL-Encoder | Qwen3-0.6B | [pg-team/pg-vision-encoder](https://huggingface.co/pg-team/pg-vision-encoder) |
## ๐ Main Results
xxx
## Citation
If you find PenguinVL useful for your research and applications, please cite using this BibTeX:
```bibtex
...
```