File size: 1,545 Bytes
86096d3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | # Gemma3-270m-VLM (Pi0.6)
A Vision-Language Model combining:
- **Vision Tower**: SigLIP from google/gemma-3-4b-pt (417M params)
- **Multi-modal Projector**: Randomly initialized (739K params)
- **Language Model**: google/gemma-3-270m (268M params)
**Total**: 686M parameters
## Architecture
- Vision hidden size: 1152
- LLM hidden size: 640
- Vocab size: 262,208 (includes 64 image tokens)
- Image token index: 262,144
## Usage
### With LLaMAFactory
```bash
llamafactory-cli train \
--stage sft \
--model_name_or_path models/gemma3-270m-vlm-with-weights \
--template gemma3 \
--dataset mllm_demo \
--freeze_vision_tower True \
--freeze_multi_modal_projector True \
--bf16 True \
...
```
### With Transformers
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained(
"models/gemma3-270m-vlm-with-weights",
torch_dtype="bfloat16"
)
processor = AutoProcessor.from_pretrained("models/gemma3-270m-vlm-with-weights")
```
## Training Recommendations
1. **Freeze vision tower and projector initially** to train only the LLM
2. **Use small learning rate** (e.g., 5e-5 or 1e-4)
3. **Gradually unfreeze** projector after LLM converges
4. Vision tower can remain frozen if using pretrained vision encoder
## Notes
- Multi-modal projector is randomly initialized and needs training
- The model uses Gemma3 tokenizer with 262,144 base tokens + 64 image tokens
- Compatible with all Gemma3 features (sliding window attention, etc.)
|