---
base_model: Qwen/Qwen2.5-VL-7B-Instruct
library_name: peft
license: apache-2.0
tags:
- peft
- lora
- vision-language
- video
- optical-flow
- uav
- drone
- motion
- embodied-ai
- spatial-intelligence
---
# SIS-Motion
SIS-Motion is a motion-aware extension of a standard video MLLM, designed to investigate whether explicitly modeling self-related dynamics can improve the joint understanding of *space* and *self* in embodied UAV scenarios.
> **Important:** SIS-Motion is **NOT a state-of-the-art model.** It is a **controlled experiment** where the only difference from the visual-only backbone is the addition of a frozen optical-flow estimator, a motion-feature branch, and a lightweight fusion connector. All other settings — base model, training data, LoRA rank, learning rate, and optimization — are kept identical. This ensures that any observed gain can be cleanly attributed to motion-aware integration rather than differences in scale or recipe.
---
## Architecture
SIS-Motion extends Qwen2.5-VL with a motion-aware branch. Video frames are processed in parallel through a shared vision encoder — once for appearance and once for optical-flow-derived motion features — then fused via a lightweight additive connector before the language model.
The forward path is:
```text
video frames
├── Qwen2.5-VL vision encoder (frozen) ───────────────> visual tokens
└── VideoFlow MOFNet (frozen, FP32)
└── forward/backward flow [T, 4, H/8, W/8]
└── forward flow (dx, dy)
└── pseudo-images [magnitude, dx, dy]
└── Qwen2.5-VL vision encoder (shared, frozen) -> motion tokens
motion tokens -> LayerNorm -> Linear(3584, 2048) -> GELU
-> Linear(2048, 3584)
visual tokens + projected motion tokens -> Qwen2.5-VL language model with LoRA
```
### 1. Appearance branch
The standard Qwen2.5-VL video processor and frozen vision encoder produce the appearance tokens used by the base model.
### 2. Motion branch
VideoFlow MOFNet receives at least three RGB frames, resizes them to `320x480`, and predicts forward and backward optical flow at `1/8` resolution. SIS-Motion uses the forward `(dx, dy)` channels, clamps invalid/extreme values, and builds a three-channel pseudo-image `[magnitude, dx, dy]`. These pseudo-images are patchified with the Qwen2.5-VL video grid and passed through the same frozen Qwen vision encoder to produce token-aligned motion features.
### 3. Additive connector
The trained `visual_flow` connector applies LayerNorm and a two-layer MLP to each motion token, then adds the result to the corresponding appearance token:
```text
motion' = Linear(GELU(Linear(LayerNorm(motion))))
fused = visual + motion'
```
The fused tokens replace the normal video-token embeddings before the language model forward pass. During inference, the LoRA adapter is merged into the base language model.
| Component | Parameters | Training state | Included here |
| --- | ---: | --- | --- |
| Qwen2.5-VL-7B base model and vision encoder | See base model | Frozen base weights; LoRA on LLM | No, downloaded separately |
| VideoFlow MOFNet | 13,453,240 | Frozen | Yes |
| `visual_flow` connector | 14,692,864 | Trained | Yes |
| LoRA adapter (`r=32`, `alpha=64`) | 20,185,088 | Trained | Yes |
| **Trainable parameters** | **34,877,952** | LoRA + connector | Yes |
| **Bundled model parameters** | **48,331,192** | Includes frozen MOFNet | Yes |
---
## Why a Controlled Experiment?
Current MLLMs show a consistent gap: they are substantially better at modeling *space* (objects, scenes, layouts) than *self* (motion, actions, agent state). To test whether explicit motion cues can alleviate this, we constructed SIS-Motion under strict controls:
- ✅ **Same base model** — Qwen2.5-VL-7B-Instruct
- ✅ **Same training data** — SIS-Motion-54K (perception + memory tasks only)
- ✅ **Same LoRA configuration** — `r=32`, `α=64`, identical target modules
- ✅ **Same optimization** — identical learning rate, schedule, and batch size
- ⬜ **Only difference** — the optical-flow-driven motion branch + additive connector
The visual-only SFT baseline serves as the direct comparison point. Any difference in results can be interpreted as the marginal contribution of motion-aware modeling.
---
## Training Configuration
| Setting | Value |
| ---------------- | ---------------------------------------------- |
| Base model | `Qwen/Qwen2.5-VL-7B-Instruct` |
| Training data | SIS-Motion-54K (MCQ + Open QA + Description) |
| Data scope | Perception & memory tasks (no reasoning tasks) |
| LoRA rank `r` | 32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.05 |
| LoRA targets | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| Optimizer | AdamW |
| Precision | BF16 |
| Trainable params | 34,877,952 (20,185,088 LoRA + 14,692,864 connector) |
| Frozen params | Visual encoder, VideoFlow optical-flow estimator, LLM base (~7B) |
---
## Results
### SIS-Bench
| Model | Spatial Avg | Self Avg | Overall |
| ------------------------ | ----------- | -------- | -------- |
| ZeroShot (Qwen2.5-VL 3B) | 65.8 | 40.5 | 53.6 |
| SFT visual-only | 72.0 | 60.3 | 66.4 |
| **SIS-Motion** | **74.2** | **63.7** | **69.1** |
Motion-aware modeling improves both spatial cognition (+2.2) and self-awareness (+3.4) over the visual-only SFT baseline. The gains are most pronounced on memory-level tasks, where temporal integration matters most.
### Downstream Transfer — Zero-Shot UAV Navigation
Evaluated on the [OpenUAV-QA](https://huggingface.co/datasets/choucsan/OpenUAV-QA) navigation benchmark without any task-specific fine-tuning:
| Model | Accuracy |
| ------------------------ | --------- |
| Qwen2.5-VL 3B (backbone) | 71.2% |
| **SIS-Motion** | **92.2%** |
The +21.0 point gain demonstrates that motion-aware representations transfer beyond the original benchmark to practical UAV decision-making scenarios.
> **Note on reported results:** The results above are from the paper's **3B** backbone (Qwen2.5-VL-3B-Instruct). This repository hosts a **7B** variant (Qwen2.5-VL-7B-Instruct) with the same architecture and training protocol. The 3B model can be reproduced using the open-source [training code](https://github.com/IntelliSensing/Self-in-Space). We release the 7B version here as a stronger off-the-shelf checkpoint; the controlled-experiment conclusion (motion-aware gains over visual-only SFT) holds across both scales.
---
## Repository Files
```text
sis_motion_config.json Architecture manifest (connector type, paths)
adapter_config.json LoRA configuration (r=32, alpha=64)
adapter_model.safetensors LoRA weights (20,185,088 params)
connector_weights.pt Motion fusion MLP weights (14,692,864 params)
MOF_kitti.pth Frozen VideoFlow MOFNet checkpoint (13,453,240 params)
images/poster.jpeg Model poster
```
**Note:** The Qwen2.5-VL-7B-Instruct base weights are **NOT included** in this repository. They are downloaded automatically from HuggingFace at inference time.
---
## Usage
This model package is not a standalone Transformers architecture. It must be loaded through the SIS-Motion evaluator, which constructs the motion branch before applying the PEFT adapter.
### 1. Clone the codebase
```bash
git clone https://github.com/IntelliSensing/Self-in-Space.git
cd Self-in-Space
```
### 2. Create the motion environment
With Conda:
```bash
bash scripts/setup_conda.sh motion
conda activate sis-motion-motion
```
Or with uv:
```bash
bash scripts/setup_uv.sh motion
source .venv-motion/bin/activate
```
The motion environment uses Python 3.10, PyTorch 2.6 with CUDA 12.4, Transformers 4.57.1, PEFT 0.18.1, and FlashAttention 2.7.4.
### 3. Prepare SIS-Bench
Place SIS-Bench under the registered data root or override the paths directly:
```text
data/SIS-Bench/SIS-Bench.jsonl
data/SIS-Bench/frames/
```
### 4. Run inference
The evaluator automatically downloads this model from HuggingFace, reads `sis_motion_config.json`, loads the VideoFlow optical-flow estimator (`MOF_kitti.pth`), applies the LoRA adapter, loads `connector_weights.pt`, and runs the full appearance-plus-motion pipeline:
```bash
MODEL_PATH=choucsan/SIS-Motion \
bash scripts/eval_motion.sh
```
An already downloaded model directory is also accepted:
```bash
MODEL_PATH=/path/to/SIS-Motion \
DATA_FILE=/path/to/SIS-Bench.jsonl \
FRAMES_DIR=/path/to/SIS-Bench/frames \
bash scripts/eval_motion.sh
```
The Qwen2.5-VL-7B base model is fetched from the `base_model_name_or_path` stored in `adapter_config.json`; it is not duplicated in this repository.
> **Note on the optical flow estimator:** The default checkpoint (`MOF_kitti.pth`) is a VideoFlow MOFNet pretrained on KITTI. Replacing it with another optical-flow estimator (e.g., RAFT or GMFlow) requires adapting the optical-flow loading and output-conversion logic. The connector weights may not transfer to a different flow backbone and would likely require retraining.
---
## Limitations
- **Not a SOTA model.** SIS-Motion is a controlled research prototype built on a 7B backbone. Stronger base models would likely yield higher absolute scores under the same setup.
- **Optical flow estimation.** The VideoFlow optical-flow estimator runs on each input video, adding compute not required by the visual-only baseline.
- **Training scope.** SIS-Motion-54K covers perception and memory tasks. Reasoning-level improvements come from generalization, not direct supervision.
- **Inference cost.** The runtime adds 28,146,104 module parameters over the base path (13,453,240 frozen MOFNet + 14,692,864 connector). The 20,185,088 LoRA parameters are merged into the base model before generation.
---
## Contact
For questions or collaboration requests:
[choucisan@gmail.com](mailto:choucisan@gmail.com)