README.md

---
language: en
license: apache-2.0
tags:
- vision
- vision-language
- knowledge-distillation
- egocentric-qa
- qwen2-vl
- edge-deployment
- model-compression
datasets:
- custom-egocentric-qa
metrics:
- latency
- throughput
- vram-usage
model-index:
- name: FirstSight-Qwen2-VL-2B-Distilled
  results:
  - task:
      type: visual-question-answering
      name: Egocentric Visual Question Answering
    metrics:
    - type: compression-ratio
      value: 3.75
      name: Model Compression
    - type: speedup
      value: 5.16
      name: Inference Speedup
    - type: vram-reduction
      value: 67.9
      name: VRAM Reduction (%)
---

# FirstSight: Distilled Qwen2-VL-2B for Efficient Egocentric QA

## Model Description

FirstSight is a **knowledge-distilled vision-language model** optimized for efficient egocentric question answering on edge devices. This model is distilled from [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) using advanced distillation techniques, achieving **3.75× compression** with minimal performance degradation.

### Key Highlights

- 🚀 **5.16× faster inference** than teacher model
- 💾 **67.9% VRAM reduction** (9.39 GB savings)
- 📦 **73.4% smaller model size** (2.21B vs 8.29B parameters)
- ⚡ **Optimized for edge deployment** on resource-constrained devices
- 🎯 **Specialized for egocentric scenarios** (first-person perspective)

### Model Architecture

- **Base Model**: [Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct)
- **Teacher Model**: [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)
- **Student Parameters**: 2.21B
- **Precision**: BFloat16 mixed precision
- **Distillation Method**: Logit-based knowledge distillation with KL divergence

## Training Details

### Training Data

- **Dataset**: Synthetic egocentric QA dataset with 5,000 training samples
- **Validation Set**: 1,000 samples
- **Question Types**: Object recognition, spatial reasoning, action understanding, temporal queries, environment understanding
- **Scenarios**: Kitchen, living room, office, outdoor, workshop

### Training Procedure

- **Framework**: PyTorch with Hugging Face Transformers
- **Epochs**: 10
- **Batch Size**: 2 per GPU with 4× gradient accumulation
- **Learning Rate**: 1e-5 (AdamW optimizer)
- **Scheduler**: Cosine annealing with 100 warmup steps
- **Loss Function**: Weighted combination of distillation loss (α=0.7) and hard label loss (α=0.3)
- **Temperature**: 2.0 for knowledge distillation
- **Hardware**: NVIDIA Quadro RTX 8000 (48GB)
- **Training Time**: ~4 hours

### Training Hyperparameters

```python
{
    "learning_rate": 1e-5,
    "optimizer": "AdamW",
    "weight_decay": 0.01,
    "gradient_accumulation_steps": 4,
    "max_grad_norm": 1.0,
    "warmup_steps": 100,
    "temperature": 2.0,
    "alpha": 0.7,
    "epochs": 10
}
```

## Performance Metrics

### Inference Speed

| Metric | Teacher (7B) | Student (2B) | Improvement |
|--------|-------------|-------------|-------------|
| **Avg Latency** | 1.260s | 0.244s | **5.16×** |
| **Throughput (samples/s)** | 0.79 | 4.10 | **5.16×** |
| **Throughput (tokens/s)** | 34.33 | 162.55 | **5.16×** |

### Memory Usage

| Metric | Teacher (7B) | Student (2B) | Savings |
|--------|-------------|-------------|---------|
| **Model Size** | 8.29B params | 2.21B params | **73.4%** |
| **Peak VRAM** | 13.81 GB | 4.43 GB | **9.39 GB** |
| **VRAM Reduction** | - | - | **67.9%** |

### Model Compression

- **Compression Ratio**: 3.75×
- **Parameter Reduction**: 73.4%
- **From**: 8.29B parameters
- **To**: 2.21B parameters

## Usage

### Installation

```bash
pip install transformers torch pillow qwen-vl-utils
```

### Inference Example

```python
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from PIL import Image
import torch

# Load model and processor
model_name = "YOUR_USERNAME/firstsight-qwen2-vl-2b-distilled"
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_name)

# Prepare image and question
image = Image.open("egocentric_image.jpg")
question = "What object am I holding in my right hand?"

# Create conversation template
messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant."
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": f"Question: {question}\nAnswer concisely:"}
        ]
    }
]

# Prepare inputs
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True)
inputs = inputs.to(model.device)

# Generate answer
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False
    )

# Decode response
response = processor.batch_decode(
    outputs[:, inputs['input_ids'].shape[1]:],
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)[0]

print(f"Answer: {response}")
```

## Intended Use

### Primary Use Cases

- **Egocentric Visual Question Answering**: Answer questions about first-person perspective images/videos
- **Edge Device Deployment**: Run VLM inference on resource-constrained hardware (mobile, IoT, AR/VR)
- **Real-time Assistive Systems**: Power low-latency visual assistants for wearable cameras
- **Smart Glasses Applications**: Enable efficient VLM capabilities on AR/VR headsets

### Supported Question Types

1. **Object Recognition**: "What object did I just pick up?"
2. **Spatial Reasoning**: "Where is the nearest door?"
3. **Action Understanding**: "What action am I performing?"
4. **Temporal Queries**: "What was I looking at 5 seconds ago?"
5. **Environment Understanding**: "What room am I in?"
6. **Counting**: "How many items are on the table?"
7. **Attribute Recognition**: "What color is the object I'm holding?"

## Limitations

- Model is specialized for **egocentric scenarios** and may perform worse on third-person images
- Trained on **synthetic data** - real-world performance may vary
- **No multimodal training** - relies solely on knowledge distillation
- May inherit biases from the teacher model (Qwen2-VL-7B)
- Limited to **short-form QA** - not optimized for long conversations

## Ethical Considerations

- **Privacy**: Egocentric images often contain sensitive personal information. Ensure proper consent and data protection.
- **Bias**: Model may exhibit biases from training data and teacher model. Evaluate on diverse datasets.
- **Misuse**: Could be used for unauthorized surveillance. Deploy responsibly with user consent.

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{firstsight2024,
  title={FirstSight: Efficient Knowledge Distillation for Vision-Language Models on Edge Devices},
  author={NYU HPML Project Team},
  year={2024},
  howpublished={\url{https://huggingface.co/YOUR_USERNAME/firstsight-qwen2-vl-2b-distilled}},
  note={Distilled from Qwen2-VL-7B-Instruct for egocentric question answering}
}
```

## Model Card Authors

NYU High Performance Machine Learning (HPML) Project Team

## Model Card Contact

For questions or feedback, please open an issue on the [GitHub repository](https://github.com/rahils/firstsight).

---

**Training Date**: December 8-9, 2024  
**Evaluation Date**: 2025-12-09T00:32:52.669538  
**Framework**: PyTorch 2.3.0, Transformers 4.57.3, BitsAndBytes 0.48.2