File size: 4,566 Bytes
fa5b6f6 f476618 fa5b6f6 f476618 fa5b6f6 f476618 fa5b6f6 f476618 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | ---
license: apache-2.0
base_model: Lightricks/LTX-2.3-22B
library_name: diffusers
---
# LTX-2.3 — 360° Equirectangular Outpainting IC-LoRA · **v0.1**
**Proof-of-concept** IC-LoRA adapter for [Lightricks/LTX-2.3-22B](https://huggingface.co/Lightricks) that
outpaints standard widescreen footage into a full **360° equirectangular** projection for immersive/VR viewing.
> This is an **early v0.1 release**. Expect rough edges, limited subject variety, and inconsistent
> coherence outside the sweet spot described below. A v0.2 with a much larger, more diverse
> dataset is planned.
---
## What it does
- **Input**: a flat 2.39:1 (cinemascope) clip and a matching equirectangular *reference* (the input
projected into the equirect canvas, with the unknown regions left masked/black)
- **Output**: the model fills in the masked regions, turning the flat shot into a plausible
360° equirectangular video that can be viewed in a VR/360 player
Intended for transforming existing live-action or cinematic footage into immersive content.
## Sweet spot (v0.1)
The v0.1 model was tuned toward a deliberately narrow domain to validate the approach:
- **Semi-static establishing city / urban scenes** (no heavy camera motion)
- **~100° horizontal field of view** in the source clip
- **2.39:1 source aspect** (standard cinemascope)
- **1024×512 @ 24 fps, 41 frames** at inference
It will **generalize poorly** outside these conditions — fast action, extreme close-ups, heavily
stylised imagery, or very different FOVs are not reliably handled yet.
## Files
| File | What it is |
|---|---|
| `ltx-2.3-22b-ic-lora-360-equirect-poc-step3500.safetensors` | The LoRA weights (final step 3500 checkpoint, ~1.3 GB) |
| `Equirect-Outpaint.json` | Reference ComfyUI workflow wired end-to-end for this LoRA |
| `samples/clipN-fl-eq.mp4` | Flat input + equirect output side-by-side (3626×960) |
| `samples/clipN-eq.mp4` | Raw equirectangular output (1920×960) — load in a 360° player |
| `samples/clipN-sweep.mp4` | 2D camera sweep through the 360° output (1920×1080) for quick preview without a VR player |
Three sample clips (`clip1`, `clip2`, `clip3`) are included under `samples/`.
## Usage
Load on top of `ltx-2.3-22b-dev.safetensors` with the LTX-2 `video_to_video` pipeline and pass:
- **Trigger word**: `equirectangular` (required, include in every prompt)
- **Reference video**: your source clip projected into the equirect canvas with unknown regions masked
- **Resolution**: 1024×512, 41 frames, 24 fps (other shapes untested)
Recommended starting points:
- LoRA strength: **1.0**
- Guidance scale (CFG): **4.0**
- STG scale: **1.0**, blocks `[29]`, mode `stg_v`
- Inference steps: **20–30**
### Companion tooling
A small ComfyUI helper pack — **[ComfyUI-EquirectProjector](https://github.com/Burgstall-labs/ComfyUI-EquirectProjector)** —
was written alongside this LoRA to produce the masked equirectangular reference from a flat clip.
Pair it with the standard LTX-2 video-to-video nodes. The `Equirect-Outpaint.json` workflow in this
repo shows the exact wiring.
## Training (v0.1)
| | |
|--|--|
| Base model | LTX-2.3-22B (dev) |
| Strategy | IC-LoRA (`video_to_video`) |
| Rank / alpha | 128 / 128 |
| Target modules | video self+cross attention + FFN |
| Resolution | 1024×512, 41 frames @ 24 fps |
| Optimizer | Prodigy (D-Adaptation), lr=1.0, constant |
| Precision | bf16, gradient checkpointing |
| Steps | 3500 |
| Hardware | 1× NVIDIA H100 80GB |
| Dataset | Small curated POC set (not released) — semi-static city establishing clips |
The final **step 3500** checkpoint is shipped here. Intermediate checkpoints were used for
validation during training but aren't included in this release.
## What's next (v0.2)
v0.2 is planned on a significantly larger and more diverse dataset (thousands of clips) covering:
- Broader subject matter (interiors, landscapes, crowds, vehicles, …)
- Varied input FOVs and focal lengths
- A wider range of camera motion — not just static establishing shots
- Better handling of the polar regions (top/bottom caps of the equirect canvas)
## Limitations
- Does not model the top/bottom caps of the sphere well — expect stretching or repetition
- Struggles with busy motion and fast cuts
- Prompt adherence is weak; conditioning is dominated by the reference video
- Outputs are not a substitute for natively captured 360 footage — this is a creative
re-projection, not a reconstruction
## License
Apache-2.0. Inherits any base-model conditions from LTX-2.3-22B.
|