Qwen2-VL-7B-DriveLM-LoRA

A LoRA adapter that fine-tunes Qwen/Qwen2-VL-7B-Instruct on the DriveLM-nuScenes v1.1 graph visual question-answering dataset. Turns the base Qwen2-VL into a driving perception + prediction + planning assistant that reads a front-view dashcam image and answers structured driving questions in the DriveLM convention (Perception / Prediction / Planning / Behavior).

TL;DR

Metric on 100 held-out val QAs (scene-based 90/10 split, seed=42)	Base Qwen2-VL-7B (zero-shot)	This adapter	Δ
Exact match	9 / 100	62 / 100	+53
Fuzzy substring match	11 / 100	65 / 100	+54
Verbose (pred > 3× GT length)	14 / 100	0 / 100	-14

Per DriveLM level (25 QAs each):

Level	Zero-shot exact	Adapter exact	Δ
Perception	5	16	+11
Prediction	4	22	+18
Planning	0	13	+13
Behavior	0	11	+11

Intended use

• Front-view dashcam / camera image → driving-focused Q&A in the DriveLM style.
• Zero-shot on driving datasets that share the DriveLM schema.
• Research + educational; not a production driving policy.

Non-intended use

• Multi-camera fusion (this LoRA was trained on a single CAM_FRONT view; sideways/rear questions filtered out during training).
• Speed estimation from a single frame (a documented ceiling — see "Known limitations" below).
• Any safety-critical driving decisions.

How to use

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info
import torch

BASE = "Qwen/Qwen2-VL-7B-Instruct"
ADAPTER = "zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA"

model = Qwen2VLForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda",
    attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

processor = AutoProcessor.from_pretrained(BASE,
    min_pixels=100*28*28, max_pixels=512*28*28)

messages = [
    {"role": "system",
     "content": "You are a driving perception assistant. You see the front "
                "camera view of the ego vehicle. Answer the question concisely "
                "based only on what is visible."},
    {"role": "user", "content": [
        {"type": "image", "image": "path/to/front_cam.jpg"},
        {"type": "text",  "text": "Predict the behavior of the ego vehicle."},
    ]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs,
                   padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=192, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:],
                             skip_special_tokens=True)[0])

Expected style of output on driving images:

• Perception: "There are two cars and one truck to the front of the ego car."
• Prediction: "Yes." / "The ego vehicle."
• Planning: "Keep going at the same speed."
• Behavior: "The ego vehicle is going straight. The ego vehicle is driving slowly."

Training details

Base model

• Qwen/Qwen2-VL-7B-Instruct (8.3B total: 675M DFN-ViT + 7.6B Qwen2 LLM).
• Vision encoder + 2×2 vision-merger MLP kept frozen.

Dataset

• OpenDriveLab/DriveLM-nuScenes v1.1 train split (696 scenes, 4,072 keyframes, 377,956 QAs).
• Scene-based 90/10 split (626 train scenes, 70 val scenes) with seed=42.
• Filtered to QAs answerable from CAM_FRONT alone (drops references to non-front cameras and back-zone questions). 154,363 train QAs after filter.

LoRA config

LoraConfig(
    r=32, lora_alpha=64, lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)

Trainable parameters: 80,740,352 / 8,372,115,968 ≈ 0.96 %.

Optimization

• 1 epoch, effective batch size 8 (per_device_batch_size=1, gradient_accumulation_steps=8).
• optim="adamw_8bit" (bitsandbytes) + bf16=True + gradient_checkpointing=True.
• Cosine schedule, lr=2e-5, warmup_ratio=0.03, weight_decay=0.
• Vision preprocessing: max_pixels=512*28*28 (~480 vision tokens per image), max_length=1024.
• attn_implementation="sdpa".

Compute

• 1× NVIDIA L40S 48 GB (AWS EC2, CUDA 13.2, driver 595.64).
• 19 h 33 min wall time. ~17 GB VRAM peak, 3.6 s/step.
• Final train_loss ≈ 0.30 (epoch avg), eval_loss ≈ 0.18 (plateau by 52 % of epoch).

Data scaling curve

Train QAs	Wall time	Val exact	Val fuzzy
500 (smoke, 3.2 ep)	12 min	40	40
20,000 (1 ep)	2 h 33 min	56	59
154,363 (1 ep, this release)	19 h 33 min	62	65

Marginal cost/benefit degrades sharply past 20 k: 7.7× more compute buys +6 exact points. If you re-train on new driving data, 20 k QAs is likely the sweet spot for the first pass.

Known limitations

1. Speed judgment: single-frame training has no temporal signal, so "driving fast / slowly / normally" predictions in the Behavior category rely on static visual cues (traffic density, road curvature). Behavior exact match tops out around 44 % (11 / 25 in the eval). Real temporal input (multi-frame from nuScenes) is expected to lift this further.
2. Side / rear scenes: this adapter was intentionally trained on the front camera only; queries about "to the back" / "back-left/right" / "behind" will hallucinate. A 3-camera panorama variant was attempted and did not help (dropped exact from 62 → 52 due to per-camera resolution loss).
3. DriveLM QA schema is narrow: the adapter learns DriveLM's specific short-answer conventions ("Yes." / "The ego vehicle." / "Please proceed."). It may under-generate on other datasets that expect longer, unstructured answers.
4. English-only: DriveLM QAs are English; Chinese/other-language driving QAs are out of distribution (though the Qwen2 base has multilingual capabilities the adapter did not train them on driving).

Reproducibility

Training pipeline: collator with token-search label masking (<|im_start|>assistant\n boundary is found in tokenized input_ids — more robust than the naive apply_chat_template prompt-length approach which suffers from BPE boundary drift).

Eval harness: 100 QAs sampled balanced across 4 DriveLM levels (25 each) from the 70 val scenes, seed=42. Metrics computed on lowercase-stripped strings.

Citation

If you use this adapter, cite the underlying works:

@article{qwen2vl,
  title={{Qwen2-VL}: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
  author={Wang, Peng and Bai, Shuai and others},
  journal={arXiv preprint arXiv:2409.12191},
  year={2024},
}

@inproceedings{drivelm,
  title={{DriveLM}: Driving with Graph Visual Question Answering},
  author={Sima, Chonghao and Renz, Katrin and others},
  booktitle={ECCV},
  year={2024},
}

License

Apache-2.0 (inherits from Qwen2-VL-7B-Instruct and DriveLM-nuScenes v1.1).