Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,245 @@
+---
+language: en
+license: apache-2.0
+tags:
+- vision
+- vision-language
+- knowledge-distillation
+- egocentric-qa
+- qwen2-vl
+- edge-deployment
+- model-compression
+datasets:
+- custom-egocentric-qa
+metrics:
+- latency
+- throughput
+- vram-usage
+model-index:
+- name: FirstSight-Qwen2-VL-2B-Distilled
+  results:
+  - task:
+      type: visual-question-answering
+      name: Egocentric Visual Question Answering
+    metrics:
+    - type: compression-ratio
+      value: 3.75
+      name: Model Compression
+    - type: speedup
+      value: 5.16
+      name: Inference Speedup
+    - type: vram-reduction
+      value: 67.9
+      name: VRAM Reduction (%)
+---
+
+# FirstSight: Distilled Qwen2-VL-2B for Efficient Egocentric QA
+
+## Model Description
+
+FirstSight is a **knowledge-distilled vision-language model** optimized for efficient egocentric question answering on edge devices. This model is distilled from [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) using advanced distillation techniques, achieving **3.75× compression** with minimal performance degradation.
+
+### Key Highlights
+
+- 🚀 **5.16× faster inference** than teacher model
+- 💾 **67.9% VRAM reduction** (9.39 GB savings)
+- 📦 **73.4% smaller model size** (2.21B vs 8.29B parameters)
+- ⚡ **Optimized for edge deployment** on resource-constrained devices
+- 🎯 **Specialized for egocentric scenarios** (first-person perspective)
+
+### Model Architecture
+
+- **Base Model**: [Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct)
+- **Teacher Model**: [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct)
+- **Student Parameters**: 2.21B
+- **Precision**: BFloat16 mixed precision
+- **Distillation Method**: Logit-based knowledge distillation with KL divergence
+
+## Training Details
+
+### Training Data
+
+- **Dataset**: Synthetic egocentric QA dataset with 5,000 training samples
+- **Validation Set**: 1,000 samples
+- **Question Types**: Object recognition, spatial reasoning, action understanding, temporal queries, environment understanding
+- **Scenarios**: Kitchen, living room, office, outdoor, workshop
+
+### Training Procedure
+
+- **Framework**: PyTorch with Hugging Face Transformers
+- **Epochs**: 10
+- **Batch Size**: 2 per GPU with 4× gradient accumulation
+- **Learning Rate**: 1e-5 (AdamW optimizer)
+- **Scheduler**: Cosine annealing with 100 warmup steps
+- **Loss Function**: Weighted combination of distillation loss (α=0.7) and hard label loss (α=0.3)
+- **Temperature**: 2.0 for knowledge distillation
+- **Hardware**: NVIDIA Quadro RTX 8000 (48GB)
+- **Training Time**: ~4 hours
+
+### Training Hyperparameters
+
+```python
+{
+    "learning_rate": 1e-5,
+    "optimizer": "AdamW",
+    "weight_decay": 0.01,
+    "gradient_accumulation_steps": 4,
+    "max_grad_norm": 1.0,
+    "warmup_steps": 100,
+    "temperature": 2.0,
+    "alpha": 0.7,
+    "epochs": 10
+}
+```
+
+## Performance Metrics
+
+### Inference Speed
+
+| Metric | Teacher (7B) | Student (2B) | Improvement |
+|--------|-------------|-------------|-------------|
+| **Avg Latency** | 1.260s | 0.244s | **5.16×** |
+| **Throughput (samples/s)** | 0.79 | 4.10 | **5.16×** |
+| **Throughput (tokens/s)** | 34.33 | 162.55 | **5.16×** |
+
+### Memory Usage
+
+| Metric | Teacher (7B) | Student (2B) | Savings |
+|--------|-------------|-------------|---------|
+| **Model Size** | 8.29B params | 2.21B params | **73.4%** |
+| **Peak VRAM** | 13.81 GB | 4.43 GB | **9.39 GB** |
+| **VRAM Reduction** | - | - | **67.9%** |
+
+### Model Compression
+
+- **Compression Ratio**: 3.75×
+- **Parameter Reduction**: 73.4%
+- **From**: 8.29B parameters
+- **To**: 2.21B parameters
+
+## Usage
+
+### Installation
+
+```bash
+pip install transformers torch pillow qwen-vl-utils
+```
+
+### Inference Example
+
+```python
+from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
+from PIL import Image
+import torch
+
+# Load model and processor
+model_name = "YOUR_USERNAME/firstsight-qwen2-vl-2b-distilled"
+model = Qwen2VLForConditionalGeneration.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
+processor = AutoProcessor.from_pretrained(model_name)
+
+# Prepare image and question
+image = Image.open("egocentric_image.jpg")
+question = "What object am I holding in my right hand?"
+
+# Create conversation template
+messages = [
+    {
+        "role": "system",
+        "content": "You are a helpful assistant."
+    },
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": image},
+            {"type": "text", "text": f"Question: {question}\nAnswer concisely:"}
+        ]
+    }
+]
+
+# Prepare inputs
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True)
+inputs = inputs.to(model.device)
+
+# Generate answer
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=128,
+        do_sample=False
+    )
+
+# Decode response
+response = processor.batch_decode(
+    outputs[:, inputs['input_ids'].shape[1]:],
+    skip_special_tokens=True,
+    clean_up_tokenization_spaces=False
+)[0]
+
+print(f"Answer: {response}")
+```
+
+## Intended Use
+
+### Primary Use Cases
+
+- **Egocentric Visual Question Answering**: Answer questions about first-person perspective images/videos
+- **Edge Device Deployment**: Run VLM inference on resource-constrained hardware (mobile, IoT, AR/VR)
+- **Real-time Assistive Systems**: Power low-latency visual assistants for wearable cameras
+- **Smart Glasses Applications**: Enable efficient VLM capabilities on AR/VR headsets
+
+### Supported Question Types
+
+1. **Object Recognition**: "What object did I just pick up?"
+2. **Spatial Reasoning**: "Where is the nearest door?"
+3. **Action Understanding**: "What action am I performing?"
+4. **Temporal Queries**: "What was I looking at 5 seconds ago?"
+5. **Environment Understanding**: "What room am I in?"
+6. **Counting**: "How many items are on the table?"
+7. **Attribute Recognition**: "What color is the object I'm holding?"
+
+## Limitations
+
+- Model is specialized for **egocentric scenarios** and may perform worse on third-person images
+- Trained on **synthetic data** - real-world performance may vary
+- **No multimodal training** - relies solely on knowledge distillation
+- May inherit biases from the teacher model (Qwen2-VL-7B)
+- Limited to **short-form QA** - not optimized for long conversations
+
+## Ethical Considerations
+
+- **Privacy**: Egocentric images often contain sensitive personal information. Ensure proper consent and data protection.
+- **Bias**: Model may exhibit biases from training data and teacher model. Evaluate on diverse datasets.
+- **Misuse**: Could be used for unauthorized surveillance. Deploy responsibly with user consent.
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{firstsight2024,
+  title={FirstSight: Efficient Knowledge Distillation for Vision-Language Models on Edge Devices},
+  author={NYU HPML Project Team},
+  year={2024},
+  howpublished={\url{https://huggingface.co/YOUR_USERNAME/firstsight-qwen2-vl-2b-distilled}},
+  note={Distilled from Qwen2-VL-7B-Instruct for egocentric question answering}
+}
+```
+
+## Model Card Authors
+
+NYU High Performance Machine Learning (HPML) Project Team
+
+## Model Card Contact
+
+For questions or feedback, please open an issue on the [GitHub repository](https://github.com/rahils/firstsight).
+
+---
+
+**Training Date**: December 8-9, 2024  
+**Evaluation Date**: 2025-12-09T00:32:52.669538  
+**Framework**: PyTorch 2.3.0, Transformers 4.57.3, BitsAndBytes 0.48.2