yuhangzang commited on
Commit
fe0ed25
·
verified ·
1 Parent(s): 14846b2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +127 -1
README.md CHANGED
@@ -21,4 +21,130 @@ library_name: transformers
21
  🤗<a href="https://huggingface.co/datasets/internlm/Spatial-SSRL-81k">Spatial-SSRL-81k Dataset</a>
22
 
23
  Spatial-SSRL-7B is a large vision-language model targeting spatial understanding, built on the base of Qwen2.5-VL-7B. It's optimized by applying Spatial-SSRL, a lightweight self-supervised reinforcement learning
24
- paradigm which can scale RLVR efficiently. The model demonstrates strong spatial intelligence while preserving the original general visual capabilities of the base model.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  🤗<a href="https://huggingface.co/datasets/internlm/Spatial-SSRL-81k">Spatial-SSRL-81k Dataset</a>
22
 
23
  Spatial-SSRL-7B is a large vision-language model targeting spatial understanding, built on the base of Qwen2.5-VL-7B. It's optimized by applying Spatial-SSRL, a lightweight self-supervised reinforcement learning
24
+ paradigm which can scale RLVR efficiently. The model demonstrates strong spatial intelligence while preserving the original general visual capabilities of the base model.
25
+
26
+ ## 📢 News
27
+ - 🚀 [2025/11/02] We have released the [🏠Spatial-SSRL Repository](https://github.com/InternLM/Spatial-SSRL).
28
+ - 🚀 [2025/11/03] We have released the [🤗Spatial-SSRL-7B Model](https://huggingface.co/internlm/Spatial-SSRL-7B),and [🤗Spatial-SSRL-81k Dataset](https://huggingface.co/datasets/internlm/Spatial-SSRL-81k).
29
+
30
+ ## 🌈 Overview
31
+ We are thrilled to introduce <strong>Spatial-SSRL</strong>, a novel self-supervised RL paradigm aimed at enhancing LVLM spatial understanding.
32
+ By optimizing Qwen2.5-VL-7B with Spatial-SSRL, the model exhibits stronger spatial intelligence across seven spatial understanding benchmarks in both image and video settings.
33
+ </p>
34
+ <p style="text-align: center;">
35
+ <img src="assets/teaser_1029final.png" alt="Teaser" width="100%">
36
+ </p>
37
+ Spatial-SSRL is a <strong>lightweight</strong> tool-free framework that is natually compatible with the RLVR training paradigm and easy to extend to a multitude of pretext tasks.
38
+ Five tasks are currently formulated in the framework, requiring only ordinary RGB and RGB-D images. <strong>And we welcome you to join Spatial-SSRL with effective pretext tasks to further strengthen the capabilities of LVLMs!</strong>
39
+
40
+ <p style="text-align: center;">
41
+ <img src="assets/pipeline_1029final.png" alt="Pipeline" width="100%">
42
+ </p>
43
+
44
+ ## 💡 Highlights
45
+ - 🔥 **Highly Scalable:** Spatial-SSRL uses ordinary raw RGB and RGB-D images instead of richly-annotated public datasets or manual labels for data curation, making it highly scalable.
46
+ - 🔥 **Cost-effective:** Avoiding the need for human labels or API calls for general LVLMs throughout the entire pipeline endows Spatial-SSRL with cost-effectiveness.
47
+ - 🔥 **Lightweight:** Prior approaches for spatial understanding heavily rely on annotation of external tools, incurring inherent errors in training data and additional cost. In constrast, Spatial-SSRL is completely tool-free and can easily be extended to more self-supervised tasks.
48
+ - 🔥 **Naturally Verifiable:** Intrinsic supervisory signals determined by pretext objectives are naturally verifiable, aligning Spatial-SSRL well with the RLVR paradigm.
49
+ <p style="text-align: center;">
50
+ <img src="assets/comparison_1029final.png" alt="Teaser" width="100%">
51
+ </p>
52
+
53
+ ## 📊 Results
54
+ We train Qwen2.5-VL-3B and Qwen2.5-VL-7B with our Spatial-SSRL paradigm and the experimental results across seven spatial understanding benchmarks are shown below.
55
+ <p style="text-align: center;">
56
+ <img src="assets/exp_result.png" alt="Pipeline" width="100%">
57
+ </p>
58
+
59
+ ## ⭐️ Quick Start
60
+ <!--To directly experience <strong>Spatial-SSRL-7B</strong>, you can try it out on huggingface (link)! -->
61
+ Here we provide a code snippet for you to start a simple trial of <strong>Spatial-SSRL-7B</strong> on your own device. You can download the model from 🤗<a href="https://huggingface.co/internlm/Spatial-SSRL-7B">Spatial-SSRL-7B Model</a > before your trial!
62
+ </p>
63
+
64
+ ```python
65
+ from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
66
+ from qwen_vl_utils import process_vision_info
67
+
68
+ model_path = "internlm/Spatial-SSRL-7B" #You can change it to your own local path if deployed already
69
+ img_path = "examples/eg1.jpg"
70
+ question = "Consider the real-world 3D locations of the objects. Which object has a higher location? A. yellow bear kite B. building"
71
+ #We recommend using the format prompt to make the inference consistent with training
72
+ format_prompt = "\n You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in \\boxed{}."
73
+
74
+ model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
75
+ model_path, torch_dtype="auto", device_map="auto"
76
+ )
77
+ processor = AutoProcessor.from_pretrained(model_path)
78
+ messages = [
79
+ {
80
+ "role": "user",
81
+ "content": [
82
+ {
83
+ "type": "image",
84
+ "image": img_path,
85
+ },
86
+ {"type": "text", "text": question + format_prompt},
87
+ ],
88
+ }
89
+ ]
90
+
91
+ text = processor.apply_chat_template(
92
+ messages, tokenize=False, add_generation_prompt=True
93
+ )
94
+ image_inputs, video_inputs = process_vision_info(messages)
95
+ inputs = processor(
96
+ text=[text],
97
+ images=image_inputs,
98
+ videos=video_inputs,
99
+ padding=True,
100
+ return_tensors="pt",
101
+ )
102
+ inputs = inputs.to("cuda")
103
+
104
+ generated_ids = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
105
+ generated_ids_trimmed = [
106
+ out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
107
+ ]
108
+ output_text = processor.batch_decode(
109
+ generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
110
+ )
111
+ print("Model Response:", output_text)
112
+ ```
113
+
114
+ ## 🛠️ Evaluation
115
+ Prepare your environment:
116
+ ```bash
117
+ git clone https://github.com/InternLM/Spatial-SSRL.git
118
+ conda create -n spatialssrl python==3.10
119
+ conda activate spatialssrl
120
+ cd Spatial-SSRL/evaluation
121
+ pip install -r requirements.txt
122
+
123
+ ```
124
+ Start your evaluation by referring to the tutorials in <a href="https://github.com/InternLM/Spatial-SSRL/blob/main/evaluation/Eval.md">Eval.md</a >
125
+
126
+ ## 👨‍💻 Todo
127
+ - [ ] Release the training code.
128
+
129
+ ## Cases
130
+ <p style="text-align: center;">
131
+ <img src="examples/case1.jpg" alt="Teaser" width="100%">
132
+ </p>
133
+ <p style="text-align: center;">
134
+ <img src="examples/case2.jpg" alt="Teaser" width="100%">
135
+ </p>
136
+
137
+ ## ✒️Citation
138
+ ```
139
+ @article{liu2025spatialssrl,
140
+ title={Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning},
141
+ author={Liu, Yuhong and Zhang, Beichen and Zang, Yuhang and Cao, Yuhang and Xing, Long and Dong, Xiaoyi and Duan, Haodong and Lin, Dahua and Wang, Jiaqi},
142
+ journal={arXiv preprint arXiv:2510.27606},
143
+ year={2025}
144
+ }
145
+ ```
146
+
147
+ ## 📄 License
148
+ ![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg) ![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)
149
+
150
+ **Usage and License Notices**: The data and code are intended and licensed for research use only.