Normalize Motius Pipeline.from_pretrained metadata
Browse files- README.md +21 -13
- model_index.json +23 -0
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
library_name:
|
| 3 |
pipeline_tag: other
|
| 4 |
tags:
|
| 5 |
- motion-generation
|
|
@@ -13,10 +13,10 @@ license: other
|
|
| 13 |
|
| 14 |
# MDM β Human Motion Diffusion Model
|
| 15 |
|
| 16 |
-
Text-to-motion baseline integrated into the
|
| 17 |
reproduction is **fully self-contained and independent of `ref_repo`**: the
|
| 18 |
network, the Gaussian-diffusion schedule, the classifier-free-guidance sampler
|
| 19 |
-
and the collate are all vendored into `
|
| 20 |
to be **bit-identical** to the released checkpoint (`max-abs-diff = 0.0` for the
|
| 21 |
same seed/input).
|
| 22 |
|
|
@@ -24,7 +24,7 @@ same seed/input).
|
|
| 24 |
|---|---|
|
| 25 |
| **Task** | Text-to-Motion (T2M) |
|
| 26 |
| **Bundle / Pipeline** | `MDMBundle` / `MDMPipeline` |
|
| 27 |
-
| **Processed HF artifact** | [`ZeyuLing/
|
| 28 |
| **Motion representation** | **HumanML3D-263** (263-dim, 20 fps, 22 joints) |
|
| 29 |
| **Text encoder** | CLIP ViT-B/32 (frozen) |
|
| 30 |
| **Paper** | *Human Motion Diffusion Model*, Tevet et al., ICLR 2023 β [arXiv:2209.14916](https://arxiv.org/abs/2209.14916) |
|
|
@@ -34,26 +34,26 @@ same seed/input).
|
|
| 34 |
|
| 35 |
## Weights
|
| 36 |
|
| 37 |
-
Current
|
| 38 |
|
| 39 |
| Artifact | Location | Contents | Status |
|
| 40 |
|---|---|---|---|
|
| 41 |
-
| MDM HumanML3D | [`ZeyuLing/
|
| 42 |
| local mirror | `checkpoints/mdm/humanml_trans_enc_512` | same layout | optional local cache |
|
| 43 |
|
| 44 |
**Use directly from the Hub:**
|
| 45 |
|
| 46 |
```python
|
| 47 |
-
from
|
| 48 |
|
| 49 |
-
pipe = MDMPipeline.from_pretrained("ZeyuLing/
|
| 50 |
motions = pipe.infer_t2m(["a person walks forward then sits down"], [120]) # list of (T, 263)
|
| 51 |
```
|
| 52 |
|
| 53 |
**Or download to disk first:**
|
| 54 |
|
| 55 |
```bash
|
| 56 |
-
huggingface-cli download ZeyuLing/
|
| 57 |
--local-dir checkpoints/mdm/humanml_trans_enc_512
|
| 58 |
```
|
| 59 |
|
|
@@ -82,7 +82,7 @@ CLIP ViT-B/32 is currently resolved by name rather than stored inside the repo.
|
|
| 82 |
| `local_vel` | 66 | local joint velocities (22Γ3) |
|
| 83 |
| `foot_contact` | 4 | binary foot-contact labels |
|
| 84 |
|
| 85 |
-
Convert to/from other spaces with `
|
| 86 |
(e.g. `hml263_to_joints`, `hml263_to_motion135`, `hml263_to_motion272`).
|
| 87 |
|
| 88 |
---
|
|
@@ -90,7 +90,7 @@ Convert to/from other spaces with `hftrainer.motion.representation.convert`
|
|
| 90 |
## Evaluation
|
| 91 |
|
| 92 |
Generation under the **official HumanML3D protocol** (standard test split, native
|
| 93 |
-
263-dim @ 20 fps, first caption) and scoring with the two persisted
|
| 94 |
evaluators. Reproduce with:
|
| 95 |
|
| 96 |
```bash
|
|
@@ -103,7 +103,7 @@ python3 scripts/eval/verify_evaluators.py --which hml263 \
|
|
| 103 |
|
| 104 |
### HumanML3D-263 evaluator (native space, n=3970)
|
| 105 |
|
| 106 |
-
| Metric |
|
| 107 |
|---|---|---|---|
|
| 108 |
| **FID** β | **0.509** | 0.544 | β
reproduced (within noise) |
|
| 109 |
| **Diversity** β | **9.563** | 9.559 | β
matches |
|
|
@@ -136,10 +136,18 @@ is the `263β272` representation mismatch.
|
|
| 136 |
|
| 137 |
## Implementation notes
|
| 138 |
|
| 139 |
-
- **Vendored, ref_repo-independent**: `
|
| 140 |
network (`network.py`), diffusion (`diffusion/`), CFG sampler and collate.
|
| 141 |
Training-only deps are stubbed (inference-only).
|
| 142 |
- **Normalization travels with the checkpoint**: `Mean.npy` / `Std.npy` are the
|
| 143 |
HumanML3D *training* stats (not the evaluator stats) and are embedded in the
|
| 144 |
artifact, eliminating the recurring "wrong Mean/Std β forward drift" bug.
|
| 145 |
- **Guidance**: classifier-free, default scale `2.5`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
library_name: motius
|
| 3 |
pipeline_tag: other
|
| 4 |
tags:
|
| 5 |
- motion-generation
|
|
|
|
| 13 |
|
| 14 |
# MDM β Human Motion Diffusion Model
|
| 15 |
|
| 16 |
+
Text-to-motion baseline integrated into the motius Model Zoo. Our
|
| 17 |
reproduction is **fully self-contained and independent of `ref_repo`**: the
|
| 18 |
network, the Gaussian-diffusion schedule, the classifier-free-guidance sampler
|
| 19 |
+
and the collate are all vendored into `motius.models.motion.mdm._mdm`, and verified
|
| 20 |
to be **bit-identical** to the released checkpoint (`max-abs-diff = 0.0` for the
|
| 21 |
same seed/input).
|
| 22 |
|
|
|
|
| 24 |
|---|---|
|
| 25 |
| **Task** | Text-to-Motion (T2M) |
|
| 26 |
| **Bundle / Pipeline** | `MDMBundle` / `MDMPipeline` |
|
| 27 |
+
| **Processed HF artifact** | [`ZeyuLing/Motius-MDM-HumanML3D`](https://huggingface.co/ZeyuLing/Motius-MDM-HumanML3D) |
|
| 28 |
| **Motion representation** | **HumanML3D-263** (263-dim, 20 fps, 22 joints) |
|
| 29 |
| **Text encoder** | CLIP ViT-B/32 (frozen) |
|
| 30 |
| **Paper** | *Human Motion Diffusion Model*, Tevet et al., ICLR 2023 β [arXiv:2209.14916](https://arxiv.org/abs/2209.14916) |
|
|
|
|
| 34 |
|
| 35 |
## Weights
|
| 36 |
|
| 37 |
+
Current motius artifact (diffusers-style `from_pretrained`):
|
| 38 |
|
| 39 |
| Artifact | Location | Contents | Status |
|
| 40 |
|---|---|---|---|
|
| 41 |
+
| MDM HumanML3D | [`ZeyuLing/Motius-MDM-HumanML3D`](https://huggingface.co/ZeyuLing/Motius-MDM-HumanML3D) | `model.safetensors` + `mdm_config.json` + `Mean.npy` / `Std.npy` | public Hub artifact; complete CLIP packaging pending |
|
| 42 |
| local mirror | `checkpoints/mdm/humanml_trans_enc_512` | same layout | optional local cache |
|
| 43 |
|
| 44 |
**Use directly from the Hub:**
|
| 45 |
|
| 46 |
```python
|
| 47 |
+
from motius.pipelines.mdm import MDMPipeline
|
| 48 |
|
| 49 |
+
pipe = MDMPipeline.from_pretrained("ZeyuLing/Motius-MDM-HumanML3D", device="cuda")
|
| 50 |
motions = pipe.infer_t2m(["a person walks forward then sits down"], [120]) # list of (T, 263)
|
| 51 |
```
|
| 52 |
|
| 53 |
**Or download to disk first:**
|
| 54 |
|
| 55 |
```bash
|
| 56 |
+
huggingface-cli download ZeyuLing/Motius-MDM-HumanML3D \
|
| 57 |
--local-dir checkpoints/mdm/humanml_trans_enc_512
|
| 58 |
```
|
| 59 |
|
|
|
|
| 82 |
| `local_vel` | 66 | local joint velocities (22Γ3) |
|
| 83 |
| `foot_contact` | 4 | binary foot-contact labels |
|
| 84 |
|
| 85 |
+
Convert to/from other spaces with `motius.motion.representation.convert`
|
| 86 |
(e.g. `hml263_to_joints`, `hml263_to_motion135`, `hml263_to_motion272`).
|
| 87 |
|
| 88 |
---
|
|
|
|
| 90 |
## Evaluation
|
| 91 |
|
| 92 |
Generation under the **official HumanML3D protocol** (standard test split, native
|
| 93 |
+
263-dim @ 20 fps, first caption) and scoring with the two persisted motius
|
| 94 |
evaluators. Reproduce with:
|
| 95 |
|
| 96 |
```bash
|
|
|
|
| 103 |
|
| 104 |
### HumanML3D-263 evaluator (native space, n=3970)
|
| 105 |
|
| 106 |
+
| Metric | motius | MDM paper | Note |
|
| 107 |
|---|---|---|---|
|
| 108 |
| **FID** β | **0.509** | 0.544 | β
reproduced (within noise) |
|
| 109 |
| **Diversity** β | **9.563** | 9.559 | β
matches |
|
|
|
|
| 136 |
|
| 137 |
## Implementation notes
|
| 138 |
|
| 139 |
+
- **Vendored, ref_repo-independent**: `motius/models/mdm/_mdm/` holds the
|
| 140 |
network (`network.py`), diffusion (`diffusion/`), CFG sampler and collate.
|
| 141 |
Training-only deps are stubbed (inference-only).
|
| 142 |
- **Normalization travels with the checkpoint**: `Mean.npy` / `Std.npy` are the
|
| 143 |
HumanML3D *training* stats (not the evaluator stats) and are embedded in the
|
| 144 |
artifact, eliminating the recurring "wrong Mean/Std β forward drift" bug.
|
| 145 |
- **Guidance**: classifier-free, default scale `2.5`.
|
| 146 |
+
|
| 147 |
+
## Direct Loading
|
| 148 |
+
|
| 149 |
+
```python
|
| 150 |
+
from motius import Pipeline
|
| 151 |
+
|
| 152 |
+
pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-MDM-HumanML3D")
|
| 153 |
+
```
|
model_index.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_class_name": "MDMPipeline",
|
| 3 |
+
"_library_name": "motius",
|
| 4 |
+
"format_version": 1,
|
| 5 |
+
"pipeline_class": "motius.pipelines.mdm.MDMPipeline",
|
| 6 |
+
"bundle_class": "motius.models.mdm.MDMBundle",
|
| 7 |
+
"tasks": [
|
| 8 |
+
"text_to_motion"
|
| 9 |
+
],
|
| 10 |
+
"required_files": [
|
| 11 |
+
"Mean.npy",
|
| 12 |
+
"Std.npy",
|
| 13 |
+
"mdm_config.json",
|
| 14 |
+
"model.safetensors"
|
| 15 |
+
],
|
| 16 |
+
"api": {
|
| 17 |
+
"loader": "motius.Pipeline.from_pretrained",
|
| 18 |
+
"task_methods": [
|
| 19 |
+
"infer_text_to_motion"
|
| 20 |
+
]
|
| 21 |
+
},
|
| 22 |
+
"artifact_format": "motius-mdm-v1"
|
| 23 |
+
}
|