internvl2-5-4b-walk-lora-v2-100

Model Description

This is a LoRA adapter for InternVL2.5-4B, fine-tuned on the WalkVLM dataset to assist visually impaired individuals with navigation hazard detection.

How to Use

Method 1: Using PEFT (Recommended)

import torch
from peft import PeftModel
from transformers import AutoModel, AutoTokenizer

# Load Base Model
base_model = AutoModel.from_pretrained(
    "OpenGVLab/InternVL2_5-4B",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("OpenGVLab/InternVL2_5-4B", trust_remote_code=True)

# Load LoRA Adapter
model = PeftModel.from_pretrained(base_model, "blind-assist/internvl2-5-4b-walk-lora-v2-100")

# Merge for faster inference (optional)
model = model.merge_and_unload()

# Use for inference
response = model.chat(
    tokenizer=tokenizer,
    pixel_values=pixel_values,  # Your preprocessed image
    question="Describe any obstacles in this scene.",
    generation_config=dict(max_new_tokens=256)
)

Method 2: Manual LoRA Merge

If PEFT doesn't work due to model architecture, use manual merging:

# See our inference script at:
# https://github.com/Blind-Assist/InternVL/blob/walkvlm/internvl_chat/test_finetuned_model.py

Training Details

• Base Model: OpenGVLab/InternVL2_5-4B
• Method: LoRA (Low-Rank Adaptation)
• LoRA Rank: 16
• Dataset: blind-assist/walk-train
• Task: Navigation hazard detection for visually impaired users

Files

• adapter_config.json - PEFT LoRA configuration
• adapter_model.safetensors - LoRA weights only (~50MB)

License

Same as base model (OpenGVLab/InternVL2_5-4B)