susanping nielsr HF Staff commited on
Commit
3b6b1bf
·
1 Parent(s): 34b81e1

Improve model card: add metadata, paper link, and sample usage (#1)

Browse files

- Improve model card: add metadata, paper link, and sample usage (4390121a0ce82ae032e2e68eeaaed9451a64ae7d)
- Resolve merge conflict with main branch (7dd7a90a0457b412f96738ba5845ad9731b4e640)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +101 -2
README.md CHANGED
@@ -1,7 +1,106 @@
1
  ---
2
  license: apache-2.0
 
3
  pipeline_tag: robotics
4
  tags:
5
- - vision-language-action
6
  - vla
7
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ library_name: transformers
4
  pipeline_tag: robotics
5
  tags:
 
6
  - vla
7
+ - vision-language-action
8
+ - real-time
9
+ ---
10
+
11
+ # Xiaomi-Robotics-0
12
+
13
+ **Xiaomi-Robotics-0** is an advanced Vision-Language-Action (VLA) model with 4.7B parameters, specifically engineered for high-performance robotic reasoning and seamless real-time execution.
14
+
15
+ - **Paper:** [Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Execution](https://huggingface.co/papers/2602.12684)
16
+ - **Project Page:** [xiaomi-robotics-0.github.io](https://xiaomi-robotics-0.github.io/)
17
+ - **Repository:** [GitHub - Xiaomi-Robotics-0](https://github.com/XiaomiRobotics/Xiaomi-Robotics-0)
18
+
19
+ ## Model Description
20
+ Xiaomi-Robotics-0 is first pre-trained on large-scale cross-embodiment robot trajectories and vision-language data, providing it with broad and generalizable action-generation capabilities. It utilizes a carefully designed training recipe and deployment strategy to address inference latency, enabling fast and smooth real-time rollouts on consumer-grade GPUs.
21
+
22
+ ## Quick Start: Deployment
23
+
24
+ The model is compatible with the Hugging Face `transformers` ecosystem. By leveraging Flash Attention 2 and bfloat16 precision, it can be run efficiently for robotic manipulation tasks.
25
+
26
+ ```python
27
+ import torch
28
+ from transformers import AutoModel, AutoProcessor
29
+
30
+ # 1. Load model and processor
31
+ model_path = "XiaomiRobotics/Xiaomi-Robotics-0"
32
+ model = AutoModel.from_pretrained(
33
+ model_path,
34
+ trust_remote_code=True,
35
+ attn_implementation="flash_attention_2",
36
+ dtype=torch.bfloat16
37
+ ).cuda().eval()
38
+ processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
39
+
40
+
41
+ # 2. Construct the prompt with multi-view inputs
42
+ language_instruction = "Pick up the red block."
43
+ instruction = (
44
+ f"<|im_start|>user
45
+ The following observations are captured from multiple views.
46
+ "
47
+ f"# Base View
48
+ <|vision_start|><|image_pad|><|vision_end|>
49
+ "
50
+ f"# Left-Wrist View
51
+ <|vision_start|><|image_pad|><|vision_end|>
52
+ "
53
+ f"Generate robot actions for the task:
54
+ {language_instruction} /no_cot<|im_end|>
55
+ "
56
+ f"<|im_start|>assistant
57
+ <cot></cot><|im_end|>
58
+ "
59
+ )
60
+
61
+ # 3. Prepare inputs
62
+ # Assuming `image_base`, `image_wrist`, and `proprio_state` are already loaded
63
+ inputs = processor(
64
+ text=[instruction],
65
+ images=[image_base, image_wrist], # [PIL.Image, PIL.Image]
66
+ videos=None,
67
+ padding=True,
68
+ return_tensors="pt",
69
+ ).to(model.device)
70
+
71
+ # Add proprioceptive state and action mask
72
+ robot_type = "libero"
73
+ inputs["state"] = torch.from_numpy(proprio_state).to(model.device, model.dtype).view(1, 1, -1)
74
+ inputs["action_mask"] = processor.get_action_mask(robot_type).to(model.device, model.dtype)
75
+
76
+ # 4. Generate action
77
+ with torch.no_grad():
78
+ outputs = model(**inputs)
79
+
80
+ # Decode raw outputs into actionable control commands
81
+ action_chunk = processor.decode_action(outputs.actions, robot_type=robot_type)
82
+ print(f"Generated Action Chunk Shape: {action_chunk.shape}")
83
+ ```
84
+
85
+ ## Benchmark Results
86
+
87
+ The model has been evaluated extensively on standard simulation benchmarks:
88
+
89
+ | Benchmark | Description | Performance |
90
+ | :--- | :--- | :--- |
91
+ | **LIBERO** | Fine-tuned on four LIBERO suites. | **98.7%** (Avg Success) |
92
+ | **CALVIN** | Fine-tuned on ABCD→D Split. | **4.80** (Avg Length) |
93
+ | **SimplerEnv** | Fine-tuned on Fractal dataset (Google Robot). | **85.5%** (VM) / **74.7%** (VA) |
94
+ | **SimplerEnv** | Fine-tuned on Bridge dataset (WidowX). | **79.2%** |
95
+
96
+ ## Citation
97
+
98
+ ```bibtex
99
+ @misc{robotics2026xiaomi,
100
+ title = {Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Execution},
101
+ author = {Xiaomi Robotics},
102
+ howpublished={\url{https://xiaomi-robotics-0.github.io}},
103
+ year = {2026},
104
+ note={Project Website}
105
+ }
106
+ ```