| --- |
| license: apache-2.0 |
| base_model: Lightricks/LTX-2.3-22B |
| library_name: diffusers |
| --- |
| |
| # LTX-2.3 — 360° Equirectangular Outpainting IC-LoRA · **v0.1** |
|
|
| **Proof-of-concept** IC-LoRA adapter for [Lightricks/LTX-2.3-22B](https://huggingface.co/Lightricks) that |
| outpaints standard widescreen footage into a full **360° equirectangular** projection for immersive/VR viewing. |
|
|
| > This is an **early v0.1 release**. Expect rough edges, limited subject variety, and inconsistent |
| > coherence outside the sweet spot described below. A v0.2 with a much larger, more diverse |
| > dataset is planned. |
|
|
| --- |
|
|
| ## What it does |
|
|
| - **Input**: a flat 2.39:1 (cinemascope) clip and a matching equirectangular *reference* (the input |
| projected into the equirect canvas, with the unknown regions left masked/black) |
| - **Output**: the model fills in the masked regions, turning the flat shot into a plausible |
| 360° equirectangular video that can be viewed in a VR/360 player |
|
|
| Intended for transforming existing live-action or cinematic footage into immersive content. |
|
|
| ## Sweet spot (v0.1) |
|
|
| The v0.1 model was tuned toward a deliberately narrow domain to validate the approach: |
|
|
| - **Semi-static establishing city / urban scenes** (no heavy camera motion) |
| - **~100° horizontal field of view** in the source clip |
| - **2.39:1 source aspect** (standard cinemascope) |
| - **1024×512 @ 24 fps, 41 frames** at inference |
|
|
| It will **generalize poorly** outside these conditions — fast action, extreme close-ups, heavily |
| stylised imagery, or very different FOVs are not reliably handled yet. |
|
|
| ## Files |
|
|
| | File | What it is | |
| |---|---| |
| | `ltx-2.3-22b-ic-lora-360-equirect-poc-step3500.safetensors` | The LoRA weights (final step 3500 checkpoint, ~1.3 GB) | |
| | `Equirect-Outpaint.json` | Reference ComfyUI workflow wired end-to-end for this LoRA | |
| | `samples/clipN-fl-eq.mp4` | Flat input + equirect output side-by-side (3626×960) | |
| | `samples/clipN-eq.mp4` | Raw equirectangular output (1920×960) — load in a 360° player | |
| | `samples/clipN-sweep.mp4` | 2D camera sweep through the 360° output (1920×1080) for quick preview without a VR player | |
|
|
| Three sample clips (`clip1`, `clip2`, `clip3`) are included under `samples/`. |
|
|
| ## Usage |
|
|
| Load on top of `ltx-2.3-22b-dev.safetensors` with the LTX-2 `video_to_video` pipeline and pass: |
|
|
| - **Trigger word**: `equirectangular` (required, include in every prompt) |
| - **Reference video**: your source clip projected into the equirect canvas with unknown regions masked |
| - **Resolution**: 1024×512, 41 frames, 24 fps (other shapes untested) |
|
|
| Recommended starting points: |
|
|
| - LoRA strength: **1.0** |
| - Guidance scale (CFG): **4.0** |
| - STG scale: **1.0**, blocks `[29]`, mode `stg_v` |
| - Inference steps: **20–30** |
|
|
| ### Companion tooling |
|
|
| A small ComfyUI helper pack — **[ComfyUI-EquirectProjector](https://github.com/Burgstall-labs/ComfyUI-EquirectProjector)** — |
| was written alongside this LoRA to produce the masked equirectangular reference from a flat clip. |
| Pair it with the standard LTX-2 video-to-video nodes. The `Equirect-Outpaint.json` workflow in this |
| repo shows the exact wiring. |
|
|
| ## Training (v0.1) |
|
|
| | | | |
| |--|--| |
| | Base model | LTX-2.3-22B (dev) | |
| | Strategy | IC-LoRA (`video_to_video`) | |
| | Rank / alpha | 128 / 128 | |
| | Target modules | video self+cross attention + FFN | |
| | Resolution | 1024×512, 41 frames @ 24 fps | |
| | Optimizer | Prodigy (D-Adaptation), lr=1.0, constant | |
| | Precision | bf16, gradient checkpointing | |
| | Steps | 3500 | |
| | Hardware | 1× NVIDIA H100 80GB | |
| | Dataset | Small curated POC set (not released) — semi-static city establishing clips | |
|
|
| The final **step 3500** checkpoint is shipped here. Intermediate checkpoints were used for |
| validation during training but aren't included in this release. |
|
|
| ## What's next (v0.2) |
|
|
| v0.2 is planned on a significantly larger and more diverse dataset (thousands of clips) covering: |
|
|
| - Broader subject matter (interiors, landscapes, crowds, vehicles, …) |
| - Varied input FOVs and focal lengths |
| - A wider range of camera motion — not just static establishing shots |
| - Better handling of the polar regions (top/bottom caps of the equirect canvas) |
|
|
| ## Limitations |
|
|
| - Does not model the top/bottom caps of the sphere well — expect stretching or repetition |
| - Struggles with busy motion and fast cuts |
| - Prompt adherence is weak; conditioning is dominated by the reference video |
| - Outputs are not a substitute for natively captured 360 footage — this is a creative |
| re-projection, not a reconstruction |
|
|
| ## License |
|
|
| Apache-2.0. Inherits any base-model conditions from LTX-2.3-22B. |
|
|
