Motion-language generation with discrete motion tokens, packaged as a Motius pipeline.
MotionGPT is the motion-language baseline from *MotionGPT: Human Motion as a Foreign Language* (Jiang et al., NeurIPS 2023). This Motius release packages the HumanML3D motion tokenizer, FLAN-T5-base-style language model with motion tokens, HumanML3D statistics, and task-facing text-to-motion / motion-to-text pipeline methods without requiring the original checkout. ## Preview | HumanML3D Sample | Input Text | SMPL Preview | | ---------------- | ---------- | ------------ | | `001840` | someone executes a roundhouse kick with their left foot. |  | | `004545` | a person jumping while raising both hands and moving apart legs. |  | | `006944` | a person moves their right hand left, right, up, and down. |  | 512px / 30fps GIF previews rendered from released HumanML3D test outputs. ## Release Snapshot | Item | Value | | ---- | ----- | | Method | MotionGPT, language modeling over text and motion tokens | | Tasks | T2M, M2T | | Venue | NeurIPS 2023 | | Motion representation | HumanML3D-263, 20 fps | | Language backbone | FLAN-T5-base-style encoder-decoder with motion tokens | | Motion tokenizer | VQ-VAE, 512-code codebook | | Checkpoint | [`ZeyuLing/Motius-MotionGPT-HumanML3D`](https://huggingface.co/ZeyuLing/Motius-MotionGPT-HumanML3D) | | Pipeline | `motius.pipelines.motiongpt.MotionGPTPipeline` | The checkpoint artifact contains `motiongpt_s3_h3d.tar`, `assets/meta/mean.npy`, `assets/meta/std.npy`, `deps/flan-t5-base/`, and `model_index.json`. ## Usage ```python from motius.pipelines.motiongpt import MotionGPTPipeline pipe = MotionGPTPipeline.from_pretrained( "ZeyuLing/Motius-MotionGPT-HumanML3D", bundle_kwargs={"local_files_only": False}, device="cuda", ) motions = pipe.infer_t2m( ["a person walks forward then sits down"], [120], ) ``` `motions` is a list of NumPy arrays. Each array has shape `(T, 263)` and is denormalized to HumanML3D physical scale. The same pipeline also exposes `infer_m2t` for captioning denormalized HumanML3D-263 motions. ```python caption = pipe.infer_m2t( [motions[0]], lengths=[len(motions[0])], )[0] ``` ## Evaluation Results Protocol: HumanML3D Official uses the selected-caption HumanML3D test protocol. MotionStreamer Evaluator and Motius Joint-Position Evaluator are computed after converting outputs through the shared SMPL-22 evaluation bridge. For FID and MM-Dist, lower is better. | Evaluator | Variant | Samples | R@1 | R@2 | R@3 | FID | MM-Dist | Diversity | Status | | --------- | ------- | ------: | --: | --: | --: | --: | ------: | --------: | ------ | | HumanML3D Official | Default | 3,962 | 0.434 | 0.600 | 0.686 | 0.156 | 3.920 | 9.747 | Measured | | MotionStreamer Evaluator | Default | 4,042 | 0.494 | 0.635 | 0.694 | 23.681 | 19.678 | 25.541 | Measured | | Motius Joint-Position Evaluator | Default | 4,034 | 0.432 | 0.580 | 0.662 | 188.125 | 38.453 | 56.885 | Measured | ### Motion-to-Text | Protocol | Samples | BLEU-4 | ROUGE-L | CIDEr | BERT F1 | R@1 | R@2 | R@3 | Matching | | -------- | ------: | -----: | ------: | ----: | -------: | --: | --: | --: | -------: | | [HumanML3D M2T](../tasks/m2t.md) | 4,400 | - | - | - | - | - | - | - | - | ## Motion Representation MotionGPT generates HumanML3D-263 features at 20 fps. Per frame: | Slice | Dim | Meaning | | ----- | --- | ------- | | `root_rot_vel` | 1 | root angular velocity | | `root_lin_vel` | 2 | root linear velocity in the horizontal plane | | `root_y` | 1 | root height | | `ric_data` | 63 | local joint positions | | `rot_data` | 126 | local joint rotations in continuous 6D format | | `local_vel` | 66 | local joint velocities | | `foot_contact` | 4 | binary foot-contact labels | The VQ-VAE converts normalized HumanML3D features into discrete motion tokens. MotionGPT then treats those tokens as a language vocabulary item alongside text tokens. ## Motius Components | Component | Path | | --------- | ---- | | Pipeline | `motius.pipelines.motiongpt.MotionGPTPipeline` | | Bundle | `motius.models.motiongpt.MotionGPTBundle` | | Runtime | `motius.models.motiongpt.network.mGPT.archs` | Only the inference-time MotionGPT modules required by the bundle are included in this public package. ## Citation ```bibtex @inproceedings{jiang2023motiongpt, title={MotionGPT: Human Motion as a Foreign Language}, author={Jiang, Biao and Chen, Xin and Liu, Wen and Yu, Jingyi and Yu, Gang and Chen, Tao}, booktitle={Advances in Neural Information Processing Systems}, year={2023} } ``` ## Direct Loading ```python from motius import Pipeline pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-MotionGPT-HumanML3D") ```