Instructions to use LiconStudio/LTX-2.3-Multiple-Subject-Reference with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use LiconStudio/LTX-2.3-Multiple-Subject-Reference with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("LiconStudio/LTX-2.3-Multiple-Subject-Reference", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -10,6 +10,33 @@ github: https://github.com/liconstudio/ComfyUI-Licon-MSR
|
|
| 10 |
library_name: diffusers
|
| 11 |
---
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
## Overview
|
| 14 |
|
| 15 |
This model implements a novel approach to multi-reference video generation using **Multiple Subject Reference (MSR)**. Instead of introducing additional encoder branches or fusion modules, we transform multiple static reference images into a pseudo-video sequence that shares the same representation space as the target video.
|
|
@@ -22,59 +49,63 @@ This LoRA requires the **[ComfyUI-Licon-MSR](https://github.com/liconstudio/Comf
|
|
| 22 |
|
| 23 |
### Multi-Reference Visual Memory
|
| 24 |
|
| 25 |
-
- **Token-level reference preservation**: Multiple reference images are encoded as video latents, preserving fine-grained visual information at token level
|
| 26 |
-
- **Native self-attention retrieval**:
|
| 27 |
-
- **In-context conditioning**: References serve as
|
| 28 |
|
| 29 |
### Flexible Reference Composition
|
| 30 |
|
| 31 |
-
- **2 to 5 reference images**: Supports varying numbers of reference inputs with increasing complexity
|
| 32 |
-
- **Complementary semantic roles**: Each reference image can
|
| 33 |
- Subject identity
|
| 34 |
-
- Object
|
| 35 |
-
- Scene
|
| 36 |
- Local textures
|
| 37 |
- Multiple viewpoints
|
| 38 |
|
| 39 |
-
|
| 40 |
## What It Can Do
|
| 41 |
|
| 42 |
### Identity Preservation Across References
|
| 43 |
|
| 44 |
-
Generate videos
|
|
|
|
| 45 |
- Multiple characters from different reference images
|
| 46 |
-
- Character
|
| 47 |
-
- Object
|
| 48 |
|
| 49 |
### Relation-Based Composition
|
| 50 |
|
| 51 |
-
Beyond
|
| 52 |
-
|
| 53 |
-
-
|
| 54 |
-
-
|
|
|
|
| 55 |
|
| 56 |
### Cross-Reference Attribute Selection
|
| 57 |
|
| 58 |
The model learns to selectively retrieve attributes from different references:
|
| 59 |
-
- Face from reference A, clothing from reference B
|
| 60 |
-
- Object identity from one reference, pose/position from another
|
| 61 |
-
- Background elements from scene references
|
| 62 |
|
| 63 |
-
|
|
|
|
|
|
|
| 64 |
|
| 65 |
-
|
| 66 |
-
- **High-motion scenes**: 50fps recommended to ensure smooth motion coherence
|
| 67 |
-
- **Generation reliability**: Typically requires 2-3 sampling runs to achieve accurate results
|
| 68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
-
##
|
| 71 |
|
| 72 |
-
###
|
| 73 |
|
| 74 |
-
|
|
| 75 |
|:---:|:---:|
|
| 76 |
-
|
|
| 77 |
-
| <img src="validition_v1/07/1.jpg" width="80"> <img src="validition_v1/07/2.jpg" width="80"> <img src="validition_v1/07/bg.png" width="80"> | [▶ Play](validition_v1/07/video.mp4) |
|
| 78 |
-
| <img src="validition_v1/05/1.png" width="70"> <img src="validition_v1/05/2.png" width="70"> <img src="validition_v1/05/5.png" width="70"> <img src="validition_v1/05/bg.png" width="70"> | [▶ Play](validition_v1/05/video.mp4) |
|
| 79 |
|
| 80 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
library_name: diffusers
|
| 11 |
---
|
| 12 |
|
| 13 |
+
# Licon MSR V2 for LTX-2.3
|
| 14 |
+
|
| 15 |
+
## What's New in V2
|
| 16 |
+
|
| 17 |
+
Compared with V1, **Licon MSR V2** introduces significant improvements in three key areas:
|
| 18 |
+
|
| 19 |
+
### 1. Improved Consistency
|
| 20 |
+
|
| 21 |
+
- Better preservation of character identity, clothing, objects, and scene details
|
| 22 |
+
- More consistent appearance across frames
|
| 23 |
+
- Improved alignment between multiple reference images and the generated video
|
| 24 |
+
- Reduced identity drift and reference attribute loss
|
| 25 |
+
|
| 26 |
+
### 2. Improved Stability
|
| 27 |
+
|
| 28 |
+
- More reliable results across repeated sampling runs
|
| 29 |
+
- Reduced visual artifacts, flickering, and temporal inconsistencies
|
| 30 |
+
- More stable generation in complex multi-subject compositions
|
| 31 |
+
- Improved handling of motion and interactions between subjects
|
| 32 |
+
|
| 33 |
+
### 3. Improved Scene Logic
|
| 34 |
+
|
| 35 |
+
- Better understanding of spatial and action relationships described in prompts
|
| 36 |
+
- More natural subject positioning and interaction
|
| 37 |
+
- Improved temporal progression from the beginning to the end of a video
|
| 38 |
+
- More coherent composition of characters, objects, and backgrounds
|
| 39 |
+
|
| 40 |
## Overview
|
| 41 |
|
| 42 |
This model implements a novel approach to multi-reference video generation using **Multiple Subject Reference (MSR)**. Instead of introducing additional encoder branches or fusion modules, we transform multiple static reference images into a pseudo-video sequence that shares the same representation space as the target video.
|
|
|
|
| 49 |
|
| 50 |
### Multi-Reference Visual Memory
|
| 51 |
|
| 52 |
+
- **Token-level reference preservation**: Multiple reference images are encoded as video latents, preserving fine-grained visual information at the token level instead of compressing them into a single embedding
|
| 53 |
+
- **Native self-attention retrieval**: Target video tokens directly access reference tokens through the model's existing self-attention mechanism, with no additional architectural components required
|
| 54 |
+
- **In-context conditioning**: References serve as visual memory within the main token sequence rather than as external conditioning inputs
|
| 55 |
|
| 56 |
### Flexible Reference Composition
|
| 57 |
|
| 58 |
+
- **2 to 5 reference images**: Supports varying numbers of reference inputs with increasing composition complexity
|
| 59 |
+
- **Complementary semantic roles**: Each reference image can provide different information:
|
| 60 |
- Subject identity
|
| 61 |
+
- Object or prop details
|
| 62 |
+
- Scene or background
|
| 63 |
- Local textures
|
| 64 |
- Multiple viewpoints
|
| 65 |
|
|
|
|
| 66 |
## What It Can Do
|
| 67 |
|
| 68 |
### Identity Preservation Across References
|
| 69 |
|
| 70 |
+
Generate videos in which multiple reference identities are simultaneously preserved:
|
| 71 |
+
|
| 72 |
- Multiple characters from different reference images
|
| 73 |
+
- Character and object combinations
|
| 74 |
+
- Object and scene compositions
|
| 75 |
|
| 76 |
### Relation-Based Composition
|
| 77 |
|
| 78 |
+
Beyond identity preservation, the model can compose references according to textual relationship descriptions:
|
| 79 |
+
|
| 80 |
+
- Action interactions, such as handing, picking up, or pushing
|
| 81 |
+
- Spatial relationships, such as left and right or foreground and background
|
| 82 |
+
- Temporal event structures, such as start → process → result
|
| 83 |
|
| 84 |
### Cross-Reference Attribute Selection
|
| 85 |
|
| 86 |
The model learns to selectively retrieve attributes from different references:
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
+
- Face from reference A and clothing from reference B
|
| 89 |
+
- Object identity from one reference and pose or position from another
|
| 90 |
+
- Background elements from scene references
|
| 91 |
|
| 92 |
+
## Usage Tips
|
|
|
|
|
|
|
| 93 |
|
| 94 |
+
- **Prompt description**: Use concise but accurate descriptions of the reference images. Both excessive and insufficient descriptions may reduce consistency.
|
| 95 |
+
- **Reference roles**: Clearly describe the role of each referenced subject, object, or scene in the target video.
|
| 96 |
+
- **High-motion scenes**: 50 fps is recommended for smoother motion coherence.
|
| 97 |
+
- **Sampling**: Complex multi-subject interactions may still benefit from multiple sampling runs.
|
| 98 |
|
| 99 |
+
## V1 vs. V2 Comparison
|
| 100 |
|
| 101 |
+
### Comparison 1
|
| 102 |
|
| 103 |
+
| V1 | V2 |
|
| 104 |
|:---:|:---:|
|
| 105 |
+
| [▶ Play V1](Validition_V2/01/V1_1.mp4) | [▶ Play V2](Validition_V2/01/V2.mp4) |
|
|
|
|
|
|
|
| 106 |
|
| 107 |
+
### Comparison 2
|
| 108 |
+
|
| 109 |
+
| V1 | V2 |
|
| 110 |
+
|:---:|:---:|
|
| 111 |
+
| [▶ Play V1](Validition_V2/02/V1.mp4) | [▶ Play V2](Validition_V2/02/V2.mp4)
|