Upload README.md with huggingface_hub

README.md

@@ -1,3 +1,241 @@
 ---
+base_model: Qwen/Qwen2.5-VL-7B-Instruct
+library_name: peft
 license: apache-2.0
+tags:
+  - peft
+  - lora
+  - vision-language
+  - video
+  - optical-flow
+  - uav
+  - drone
+  - motion
+  - embodied-ai
+  - spatial-intelligence
 ---
+
+# SIS-Motion
+
+<p align="center">
+  <img src="images/poster.jpeg" alt="SIS-Motion" width="900"/>
+</p>
+<p align="center">
+  <a href="https://arxiv.org"><img src="https://img.shields.io/badge/arXiv-Paper-b31b1b?style=for-the-badge&logo=arxiv" alt="arXiv"></a>
+  <a href="https://choucisan.github.io/publications/self-in-space"><img src="https://img.shields.io/badge/Website-Project_Page-blue?style=for-the-badge" alt="Website"></a>
+  <a href="https://github.com/IntelliSensing/Self-in-Space"><img src="https://img.shields.io/badge/GitHub-Self--in--Space-181717?style=for-the-badge&logo=github" alt="GitHub"></a>
+  <a href="https://huggingface.co/choucsan/SIS-Motion"><img src="https://img.shields.io/badge/%F0%9F%A4%97_HuggingFace-Model-yellow?style=for-the-badge" alt="Hugging Face"></a>
+  <a href="https://choosealicense.com/licenses/apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-green?style=for-the-badge" alt="License"></a>
+</p>
+
+
+SIS-Motion is a motion-aware extension of a standard video MLLM, designed to investigate whether explicitly modeling self-related dynamics can improve the joint understanding of *space* and *self* in embodied UAV scenarios.
+
+> **Important:** SIS-Motion is **NOT a state-of-the-art model.** It is a **controlled experiment** where the only difference from the visual-only backbone is the addition of a frozen optical-flow estimator, a motion-feature branch, and a lightweight fusion connector. All other settings — base model, training data, LoRA rank, learning rate, and optimization — are kept identical. This ensures that any observed gain can be cleanly attributed to motion-aware integration rather than differences in scale or recipe.
+
+---
+
+## Architecture
+
+SIS-Motion extends Qwen2.5-VL with a motion-aware branch. Video frames are processed in parallel through a shared vision encoder — once for appearance and once for optical-flow-derived motion features — then fused via a lightweight additive connector before the language model.
+
+<p align="center">
+  <img src="images/sis-motion-framework.webp" alt="SIS-Motion Framework" width="900"/>
+</p>
+
+The forward path is:
+
+```text
+video frames
+├── Qwen2.5-VL vision encoder (frozen) ───────────────> visual tokens
+└── VideoFlow MOFNet (frozen, FP32)
+    └── forward/backward flow [T, 4, H/8, W/8]
+        └── forward flow (dx, dy)
+            └── pseudo-images [magnitude, dx, dy]
+                └── Qwen2.5-VL vision encoder (shared, frozen) -> motion tokens
+
+motion tokens -> LayerNorm -> Linear(3584, 2048) -> GELU
+              -> Linear(2048, 3584)
+visual tokens + projected motion tokens -> Qwen2.5-VL language model with LoRA
+```
+
+### 1. Appearance branch
+
+The standard Qwen2.5-VL video processor and frozen vision encoder produce the appearance tokens used by the base model.
+
+### 2. Motion branch
+
+VideoFlow MOFNet receives at least three RGB frames, resizes them to `320x480`, and predicts forward and backward optical flow at `1/8` resolution. SIS-Motion uses the forward `(dx, dy)` channels, clamps invalid/extreme values, and builds a three-channel pseudo-image `[magnitude, dx, dy]`. These pseudo-images are patchified with the Qwen2.5-VL video grid and passed through the same frozen Qwen vision encoder to produce token-aligned motion features.
+
+### 3. Additive connector
+
+The trained `visual_flow` connector applies LayerNorm and a two-layer MLP to each motion token, then adds the result to the corresponding appearance token:
+
+```text
+motion' = Linear(GELU(Linear(LayerNorm(motion))))
+fused   = visual + motion'
+```
+
+The fused tokens replace the normal video-token embeddings before the language model forward pass. During inference, the LoRA adapter is merged into the base language model.
+
+| Component | Parameters | Training state | Included here |
+| --- | ---: | --- | --- |
+| Qwen2.5-VL-7B base model and vision encoder | See base model | Frozen base weights; LoRA on LLM | No, downloaded separately |
+| VideoFlow MOFNet | 13,453,240 | Frozen | Yes |
+| `visual_flow` connector | 14,692,864 | Trained | Yes |
+| LoRA adapter (`r=32`, `alpha=64`) | 20,185,088 | Trained | Yes |
+| **Trainable parameters** | **34,877,952** | LoRA + connector | Yes |
+| **Bundled model parameters** | **48,331,192** | Includes frozen MOFNet | Yes |
+
+---
+
+## Why a Controlled Experiment?
+
+Current MLLMs show a consistent gap: they are substantially better at modeling *space* (objects, scenes, layouts) than *self* (motion, actions, agent state). To test whether explicit motion cues can alleviate this, we constructed SIS-Motion under strict controls:
+
+- ✅ **Same base model** — Qwen2.5-VL-7B-Instruct
+- ✅ **Same training data** — SIS-Motion-54K (perception + memory tasks only)
+- ✅ **Same LoRA configuration** — `r=32`, `α=64`, identical target modules
+- ✅ **Same optimization** — identical learning rate, schedule, and batch size
+- ⬜ **Only difference** — the optical-flow-driven motion branch + additive connector
+
+The visual-only SFT baseline serves as the direct comparison point. Any difference in results can be interpreted as the marginal contribution of motion-aware modeling.
+
+---
+
+## Training Configuration
+
+| Setting          | Value                                          |
+| ---------------- | ---------------------------------------------- |
+| Base model       | `Qwen/Qwen2.5-VL-7B-Instruct`                  |
+| Training data    | SIS-Motion-54K (MCQ + Open QA + Description)   |
+| Data scope       | Perception & memory tasks (no reasoning tasks) |
+| LoRA rank `r`    | 32                                             |
+| LoRA alpha       | 64                                             |
+| LoRA dropout     | 0.05                                           |
+| LoRA targets     | `q_proj`, `k_proj`, `v_proj`, `o_proj`         |
+| Optimizer        | AdamW                                          |
+| Precision        | BF16                                           |
+| Trainable params | 34,877,952 (20,185,088 LoRA + 14,692,864 connector) |
+| Frozen params    | Visual encoder, VideoFlow optical-flow estimator, LLM base (~7B) |
+
+---
+
+## Results
+
+### SIS-Bench
+
+| Model                    | Spatial Avg | Self Avg | Overall  |
+| ------------------------ | ----------- | -------- | -------- |
+| ZeroShot (Qwen2.5-VL 3B) | 65.8        | 40.5     | 53.6     |
+| SFT visual-only          | 72.0        | 60.3     | 66.4     |
+| **SIS-Motion**           | **74.2**    | **63.7** | **69.1** |
+
+Motion-aware modeling improves both spatial cognition (+2.2) and self-awareness (+3.4) over the visual-only SFT baseline. The gains are most pronounced on memory-level tasks, where temporal integration matters most.
+
+### Downstream Transfer — Zero-Shot UAV Navigation
+
+Evaluated on the [OpenUAV-QA](https://huggingface.co/datasets/choucsan/OpenUAV-QA) navigation benchmark without any task-specific fine-tuning:
+
+| Model                    | Accuracy  |
+| ------------------------ | --------- |
+| Qwen2.5-VL 3B (backbone) | 71.2%     |
+| **SIS-Motion**           | **92.2%** |
+
+The +21.0 point gain demonstrates that motion-aware representations transfer beyond the original benchmark to practical UAV decision-making scenarios.
+
+> **Note on reported results:** The results above are from the paper's **3B** backbone (Qwen2.5-VL-3B-Instruct). This repository hosts a **7B** variant (Qwen2.5-VL-7B-Instruct) with the same architecture and training protocol. The 3B model can be reproduced using the open-source [training code](https://github.com/IntelliSensing/Self-in-Space). We release the 7B version here as a stronger off-the-shelf checkpoint; the controlled-experiment conclusion (motion-aware gains over visual-only SFT) holds across both scales.
+
+---
+
+## Repository Files
+
+```text
+sis_motion_config.json       Architecture manifest (connector type, paths)
+adapter_config.json          LoRA configuration (r=32, alpha=64)
+adapter_model.safetensors    LoRA weights (20,185,088 params)
+connector_weights.pt         Motion fusion MLP weights (14,692,864 params)
+MOF_kitti.pth                Frozen VideoFlow MOFNet checkpoint (13,453,240 params)
+images/poster.jpeg           Model poster
+```
+
+**Note:** The Qwen2.5-VL-7B-Instruct base weights are **NOT included** in this repository. They are downloaded automatically from HuggingFace at inference time.
+
+---
+
+## Usage
+
+This model package is not a standalone Transformers architecture. It must be loaded through the SIS-Motion evaluator, which constructs the motion branch before applying the PEFT adapter.
+
+### 1. Clone the codebase
+
+```bash
+git clone https://github.com/IntelliSensing/Self-in-Space.git
+cd Self-in-Space
+```
+
+### 2. Create the motion environment
+
+With Conda:
+
+```bash
+bash scripts/setup_conda.sh motion
+conda activate sis-motion-motion
+```
+
+Or with uv:
+
+```bash
+bash scripts/setup_uv.sh motion
+source .venv-motion/bin/activate
+```
+
+The motion environment uses Python 3.10, PyTorch 2.6 with CUDA 12.4, Transformers 4.57.1, PEFT 0.18.1, and FlashAttention 2.7.4.
+
+### 3. Prepare SIS-Bench
+
+Place SIS-Bench under the registered data root or override the paths directly:
+
+```text
+data/SIS-Bench/SIS-Bench.jsonl
+data/SIS-Bench/frames/
+```
+
+### 4. Run inference
+
+The evaluator automatically downloads this model from HuggingFace, reads `sis_motion_config.json`, loads the VideoFlow optical-flow estimator (`MOF_kitti.pth`), applies the LoRA adapter, loads `connector_weights.pt`, and runs the full appearance-plus-motion pipeline:
+
+```bash
+MODEL_PATH=choucsan/SIS-Motion \
+bash scripts/eval_motion.sh
+```
+
+An already downloaded model directory is also accepted:
+
+```bash
+MODEL_PATH=/path/to/SIS-Motion \
+DATA_FILE=/path/to/SIS-Bench.jsonl \
+FRAMES_DIR=/path/to/SIS-Bench/frames \
+bash scripts/eval_motion.sh
+```
+
+The Qwen2.5-VL-7B base model is fetched from the `base_model_name_or_path` stored in `adapter_config.json`; it is not duplicated in this repository.
+
+> **Note on the optical flow estimator:** The default checkpoint (`MOF_kitti.pth`) is a VideoFlow MOFNet pretrained on KITTI. Replacing it with another optical-flow estimator (e.g., RAFT or GMFlow) requires adapting the optical-flow loading and output-conversion logic. The connector weights may not transfer to a different flow backbone and would likely require retraining.
+
+---
+
+## Limitations
+
+- **Not a SOTA model.** SIS-Motion is a controlled research prototype built on a 7B backbone. Stronger base models would likely yield higher absolute scores under the same setup.
+- **Optical flow estimation.** The VideoFlow optical-flow estimator runs on each input video, adding compute not required by the visual-only baseline.
+- **Training scope.** SIS-Motion-54K covers perception and memory tasks. Reasoning-level improvements come from generalization, not direct supervision.
+- **Inference cost.** The runtime adds 28,146,104 module parameters over the base path (13,453,240 frozen MOFNet + 14,692,864 connector). The 20,185,088 LoRA parameters are merged into the base model before generation.
+
+---
+
+## Contact
+
+For questions or collaboration requests:
+
+[choucisan@gmail.com](mailto:choucisan@gmail.com)