RuneXX

/

LTX-2.3-2.5-Workflows

like 843

Looks amazing, but.. I can't find the node in the manager and It does not install through git pull

Unpacking objects: 100% (13/13), 13.72 KiB | 246.00 KiB/s, done.
From https://github.com/kijai/ComfyUI-PromptRelay

• branch HEAD -> FETCH_HEAD
  fatal: refusing to merge unrelated histories

Sounds like something went wrong with the git pull.
Just delete that node folder, and try git clone again... hopefully that fixes it

I fixed it. it is amazing to be honest.. thank you so much.
Just a question: can you upgrade this workflow to include a starting image for each segment?

Yeah the prompt relay thing is pretty neat, can do some really nice stuff

Just a question: can you upgrade this workflow to include a starting image for each segment?

I was actually making a workflow a bit like that (before the prompt relay came). Based a on the "music video creator" workflow from earlier. Where you set length, image and prompt per segment.

But using the Prompt Relay might also actually work... If each image is set to the exact frame start of each prompt segment.
Worth a try ;-) will do

(for simpler "stories" you can prompt for scene change already, with the Prompt Relay - see video 2 in the example above. But for a complete new camera angle, image input would give more control)

yes please, thank you.. and I agree, for simple long video with camera going back and forth, it is easy with a text prompt only.. but for a complex camera angle, a guiding image would be perfect

Gave it a try, unfortunately its not going to be 100% what you had in mind perhaps, since the frame inject node only give a new frame image. But LTX might interpret that either as a new scene, OR transition to that image (first last frame logic). But still an interesting workflow ;-) will upload in a few

depending on what you had in mind with back and forth. It might switch between say 2 persons, or it might transition/move the camera slowly between 2 persons.
With some prompting you can probably control it though. Starting the prompt for a segment with : "Scene cut/new scene" that LTX seems to understand ;-)

(the other workflow based on the "music creator" is a totally new scene at each segment - that can be different or similar to previous scene. Will upload that too, just testing out a few things for consistent audio)

works great! I could not pull this off properly with FMLF, with t2v, i2v list wf continues ..I kept trying but something would always be wrong...here it was done on 3rd attempt when prompting was dialed in properly.

😃 this was quick and dirty prompting - it took a few minutes to type it - it was not even elaborated in detail more of a rough sketch , prompt adherence is really good, alas still no Guns lora for LTX2.3...

Looks nice ;-) and yes the prompt relay is much easier to use to create a little story than FMLF

"i am an influencer..." 🤣🤣 funny video haha

I2V - Multi-Ref Image & Multi-Event prompt relay workflow
Added a workflow variant of the multi-prompt workflow where you can also input a start frame reference image for each sequence, from idea by @APCOTech ;-)

( a little bit experimental, but seems to work ok from a couple of test runs )

To create different first frame images with consistent characters you can use Flux Klein (just prompt for a new scene with same characters), Qwen Image with Next-Scene or Multiple-Angles lora, or just ask Nano Banana, Chat GPT, or Qwen Chat to do it ;-) (I'll upload a "helper" workflow with Klein and Qwen to create first frame images and there is one natively already inside ComfyUI as well)

Can try it out here:
https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Movie-Maker

( strictly speaking the simpler workflow with one reference image, can pretty much do the same. Change camera angles, scene etc with consistent characters, if both/all are in the input ref. image. But having multiple ref.images gives you even more granular control)

Hi @Tom , Is LTXCropGuild bugged? Because 8s × 4 = 32s, but your video is 31s.

no.. the fade out at the end was a bit long, so i removed a bit manually before posting ;-) it was probably 32 seconds

I2V - Multi-Ref Image & Multi-Event prompt relay workflow
Added a workflow variant of the multi-prompt workflow where you can also input a start frame reference image for each sequence, from idea by @APCOTech ;-)

( a little bit experimental, but seems to work ok from a couple of test runs )

To create different first frame images with consistent characters you can use Flux Klein (just prompt for a new scene with same characters), Qwen Image with Next-Scene or Multiple-Angles lora, or just ask Nano Banana, Chat GPT, or Qwen Chat to do it ;-) (I'll upload a "helper" workflow with Klein and Qwen to create first frame images and there is one natively already inside ComfyUI as well)

Can try it out here:
https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Movie-Maker

( strictly speaking the simpler workflow with one reference image, can pretty much do the same. Change camera angles, scene etc with consistent characters, if both/all are in the input ref. image. But having multiple ref.images gives you even more granular control)

OMG, you did it!! You are doing a fantastic job here my friend. with your workflows and @Kijai 's nodes, you are making LTX equal, if not better, than Kling or Sora..
Thank you so much <3

no.. the fade out at the end was a bit long, so i removed a bit manually before posting ;-) it was probably 32 seconds

This was driving me crazy! I thought I am doing something wrong since I am using GGUFs.. The last second always flickers and repeats many frames from the video.. I gave up and I used to cut the last second.. but now when I read you comment, I think it is the normal behavior. In the old Wan Vace, there was something like (TrimLatent) as I remember. It is not a big deal since I just cut that last second, but is there something in LTX like TrimLatent or whatever its name?

This was driving me crazy! I thought I am doing something wrong since I am using GGUFs.. The last second always flickers and repeats many frames from the video..

are you using v.1.0 of the upscaler? That had a bug with last frames, since fixed with the v1.1 of the upscale model
https://huggingface.co/Lightricks/LTX-2.3/tree/main

And if you already are on v.1.1 of the upscale model, it could be something with the gguf, i havent used gguf lately...
But more likely an ltx crop that is missing or something like that

This was driving me crazy! I thought I am doing something wrong since I am using GGUFs.. The last second always flickers and repeats many frames from the video..

are you using v.1.0 of the upscaler? That had a bug with last frames, since fixed with the v1.1 of the upscale model
https://huggingface.co/Lightricks/LTX-2.3/tree/main

And if you already are on v.1.1 of the upscale model, it could be something with the gguf, i havent used gguf lately...
But more likely an ltx crop that is missing or something like that

Well, this may sound like a stupid approach of me, but I do not use the upscaler at all. 1.0 and 1. tends to change details in the images (first and last frames). so I bypass the (down)scale node and I render my videos in 1024 width or height.. (on 4080 16gb, 7 seconds 1024x1024@24fps) in 137seconds total (13.25s/it). I know it is slower, but this way I keep the details of the images 100%..
Anyway, may be ignoring the upscale model causes this last second deformation or may be the gguf.. but the only fix is just to cut that last second..

Not a stupid approach at all ;-) sometimes 1pass is preferable (for the reasons you mentioned, and some of the newer stuff even only works with 1pass.. like HDR video lora (workflow coming))

But then you are probably missing the LTX Crop node (that is in the 2nd pass).
In the last "group" in the workflow, the "decode" group. Add a LTX Crop node before the VAE vidoe decode.

And if that all sounds greek, I'll adapt the workflow above to have a simple toggle, so that it can be used both as single pass and 2-pass (will upload asap).

The crop node removes "junk" frames at the end reserved as guider frames for LTX.
If you want to try it would be something like this:

(the only tricky part is connecting the correct positive / negative inputs (this node doesnt use the outputs other than latent since its at the end of wf)
.. if you see whats connected to the crop node in the 2nd pass, its the same positive/negative)

Sadly could not pull it off at 720p , not at this length.

Sadly could not pull it off at 720p , not at this length.

That looks great ;-)

Longer videos can be a bit demanding on the pc for sure.
I'll see if i can make some "continue from last generation" part.. so that you can say generate 20 seconds story. Then use that 20 second video as input to generate another 20 second... and so on.
The only tricky part is that the voices must be in the frames uses as reference for next 20 seconds, but should be doable ;-)

Not a stupid approach at all ;-) sometimes 1pass is preferable (for the reasons you mentioned, and some of the newer stuff even only works with 1pass.. like HDR video lora (workflow coming))

But then you are probably missing the LTX Crop node (that is in the 2nd pass).
In the last "group" in the workflow, the "decode" group. Add a LTX Crop node before the VAE vidoe decode.

And if that all sounds greek, I'll adapt the workflow above to have a simple toggle, so that it can be used both as single pass and 2-pass (will upload asap).

The crop node removes "junk" frames at the end reserved as guider frames for LTX.
If you want to try it would be something like this:

(the only tricky part is connecting the correct positive / negative inputs (this node doesnt use the outputs other than latent since its at the end of wf)
.. if you see whats connected to the crop node in the 2nd pass, its the same positive/negative)

I really never knew that this node exists.. Thank you so much, you are great and very helpful <3

lmao this turned out to be wild, totally not what I expected but hilarious anyway.

first run, prompting could be tighter but it has delivered what I asked for.

nice nice ;-) thats the tightest fight-scene i seen in LTX ever i think... looks like those frames helped a lot ;-)

here is final video with a few effects I wanted to show, anime inspired. so one thing that becomes clear with prompt relay encode node is prompt following and adherence in continuous action sequences as in the clip is good up to 20 seconds length and u have to carefully make images and action in prompt must be realistically possible to happen within time length defined as per segment. 20+ seconds length and audio will not be generated properly something will be missing, prompted audio effects are not there or dialog will not be spoken, model drifts away from prompt..15-20 sec it all works.

That looks great ;.)

and yes LTX is 10-20 seconds videos max.. for single video generations
After that weird things happen. Would even say the sweet spot is 10-15 seconds (but up to 20).

For longer than that it needs a mask and continue approach (like the other long video workflows), where its essentially multiple 15s or so videos stitched together as one.
The challenge with that is consistency. With input images as the prompt relay has, this might work more easily. But the voice consistency needs to sample from the previous videos, and all characters must talk within the last seconds of previous video, for it to work well (so the easier route then is custom audio input or audio id lora). Will try something for that, see if it can work ok ;-)

But for 20 seconds shorts, Prompt Relay is neat ;-)

That looks great ;.)

and yes LTX is 10-20 seconds videos max.. for single video generations
After that weird things happen. Would even say the sweet spot is 10-15 seconds (but up to 20).

For longer than that it needs a mask and continue approach (like the other long video workflows), where its essentially multiple 15s or so videos stitched together as one.
The challenge with that is consistency. With input images as the prompt relay has, this might work more easily. But the voice consistency needs to sample from the previous videos, and all characters must talk within the last seconds of previous video, for it to work well (so the easier route then is custom audio input or audio id lora). Will try something for that, see if it can work ok ;-)

But for 20 seconds shorts, Prompt Relay is neat ;-)

yes 20 sec if you want great prompt adherence and if it is a complex scene with action in continuous scene , still it is easy to stitch a few 20 sec long scenes made with this wf and add crossfades in D.Resolve, it takes a few minutes...however this wf works fine for longer scenes, I already posted 34-35 sec long scenes but there are scene changes in such videos they are not one continuous shot.

True, might be able to push it a bit beyond the LTX recommended max 20 since it has fresh new images at each segment. So the quality degradation will be less ...
And for sure is a fun node, that makes it much easier to create something a bit intentional instead of just rolling the dice and hoping for the best ;-)

We can still use LTX2 NAG this way ( your workflow use ZeroOut and disable )

Think NAG should work yeah. I might have left that zero out in there when initially making the workflow, to start with as few other factors as possible that could change the result.
I will test myself too. There is one caveat, that the Prompt Relay "changes" the model, and so does the NAG too (both have model in and out). So they might "crash" and not be working well together , but probably it should be fine

Interesting concept for sure. It is however (imo) fiddly as a motherf... timing the segments if you are not using a "end scene" type, is kinda awkward and makes for.. well.. a lot of fiddling. Since LTXV seems sometimes dumb as... well.. its not too easy to make it follow the prompt closely either 😀

Anyway.. was fairly well, although the bartender is the brother of Harry Potter and magically conjure the glass on the counter, well 😏

Looks cool ;-) yes it can be a few awkward silences with segments.. but its all good fun, with a bit more control over the direction of the video ;-)

went for more cinematic era look with prompted high-contrast chiaroscuro lighting in master prompt. This is with reasoning v3 lora at 0.75 and fight V2 lora, tried bumping to 30fps as a fix to blurring and smearing because of fast action but it helps only a bit, topaz post processing helped more to reduce it. That hat turning to dust at the end needs prompt anchoring, it was not prompted at all that it should go flying so model just disintegrated it after it fell on the floor :D

That looks pretty good ;-) nice nice

Comfy just added a RIFE and FILM frame interpolation node, that is doubling the FPS. You might want to try it out

And there is also RTX Super Resolution node (insanely fast on RTX cards), FlashVSR and SeedVR2
(if you want to try something in comfy, instead of Topaz. Not sure if they are quite as good as Topaz, but at least some alternatives ;-))

That looks pretty good ;-) nice nice

Comfy just added a RIFE and FILM frame interpolation node, that is doubling the FPS. You might want to try it out

And there is also RTX Super Resolution node (insanely fast on RTX cards), FlashVSR and SeedVR2
(if you want to try something in comfy, instead of Topaz. Not sure if they are quite as good as Topaz, but at least some alternatives ;-))

Any chance for integration of Kiaji's new Vid2Vid extension node with an on off switch in this workflow, I haven't had a chance to play with it yet so not sure how it works... I did lots of nice tests with your vid2vid extension workflows that produced stunning results, I guess this node that Kijai made (LTXVAudioVideoMask) can be now integrated in any workflow? RTX Superresolutions is nuts, it is blazing fast on 5000 series.

Kiaji's new Vid2Vid extension node

Not sure what node that is (Vid2Vid). Or did you mean the LTXVAudioVideoMask node?

If you mean LTXVAudioVideoMask it could be integrated, it basically adds a mask for LTX to "inpaint". So that could either be before a video, in the middle of a video, or at end of video (extending/padding).
You want to extend on the video generated with prompt relay?

Kiaji's new Vid2Vid extension node

Not sure what node that is (Vid2Vid). Or did you mean the LTXVAudioVideoMask node?

If you mean LTXVAudioVideoMask it could be integrated, it basically adds a mask for LTX to "inpaint". So that could either be before a video, in the middle of a video, or at end of video (extending/padding).
You want to extend on the video generated with prompt relay?

yeah LTXVAudioVideoMask, I was not even aware it was new thing until yesterday! I haven't worked with that node, I think I used older vid2vid wfs u made that did not use it, I just saw post u made with Joker vid that uses that node. I am not sure how it works I need to "dissect" a workflow to learn what it does and where it connects to be able to include it in other wfs, so far I don't know that we have publicly available (even grok agrees) I2V or T2V workflow for LTX2.3 that integrates vid to vid extension with LTXVAudioVideoMask after first video gets generated in T2V or I2V mode, the only such wf that exists seems to be behind patreon paywall made by Nerdy Rodent.

I don't know that we have publicly available (even grok agrees) I2V or T2V workflow for LTX2.3 that integrates vid to vid extension with LTXVAudioVideoMask

Its used in many of the workflows here: https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Video-2-Video

• Extend Video uses LTXVAudioVideoMask to pad extended frames to any video input
• Foley uses LTXVAudioVideoMask to mask the audio, so you can add LTX audio to any silent video input (from wan etc)
• Re-Take uses LTXVAudioVideoMask to mask part of a video to re-shoot/re-take that section
• Just-Talk uses LTXVAudioVideoMask plus masking the face, to add lip-sync to any silent video

But if you had a particular other use of LTXVAudioVideoMask I can try make it.. Its a bit multi-use this node since its masking masking the video/audio.. can be used for many things ;-)

What did Nerdy Rodent use it for? Seen his youtubes, funny guy ... ;-)

Kiaji's new Vid2Vid extension node

Not sure what node that is (Vid2Vid). Or did you mean the LTXVAudioVideoMask node?

If you mean LTXVAudioVideoMask it could be integrated, it basically adds a mask for LTX to "inpaint". So that could either be before a video, in the middle of a video, or at end of video (extending/padding).
You want to extend on the video generated with prompt relay?

yeah LTXVAudioVideoMask, I was not even aware it was new thing until yesterday! I haven't worked with that node, I think I used older vid2vid wfs u made that did not use it, I just saw post u made with Joker vid that uses that node. I am not sure how it works I need to "dissect" a workflow to learn what it does and where it connects to be able to include it in other wfs, so far I don't know that we have publicly available (even grok agrees) I2V or T2V workflow for LTX2.3 that integrates vid to vid extension with LTXVAudioVideoMask after first video gets generated in T2V or I2V mode, the only such wf that exists seems to be behind patreon paywall made by Nerdy Rodent.

yeah he has funny entertaining delivery in his videos they are fun to watch and easy to learn from, basically he made simple wf T2V or I2V with integrated vid2vid extension after first generation, he did multiple test to compare GGUF vs FP8 in terms of quality...

something that I could recreate , after I figure out all set and get nodes :D

Basically looks like he is extending a 5 second video to 10 (or 4.85s to 9s to be exact, since he is doing 25fps).
The LTXAudioVideo mask, in pad mode, extends the video. You set the start from ... and end at. The end at can be beyond the duration of the input video, and by that padding masks, that LTX inpaints.
Since it uses some of the input video as reference, it can work great, with consistent movements and audio

Pretty much looks like same as my "Extend Any Video" workflow, although he is just using 1-pass (but i think in my version there is an easy toggle to set 1-pass if one prefer)
If you try the LTX-2.3-V2V_Extend_Any_Video.json here https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Video-2-Video

But I'll check out his rodent video and see if its something else ;-)

Actually might be a good idea for the Prompt Relay workflow.
Instead of starting from scratch with the very first image, use a video as input instead. And then continue on from there with the prompt relay stuff setting timer and frame images for the following segments.

And maybe whats what you meant ;-) will try something like that.

Extend after the prompt relay stuff, could also be done, but thats pretty much what "Extend Any Video" already do. Takes a video input and extends it

Actually might be a good idea for the Prompt Relay workflow.
Instead of starting from scratch with the very first image, use a video as input instead. And then continue on from there with the prompt relay stuff setting timer and frame images for the following segments.

And maybe whats what you meant ;-) will try something like that.

Extend after the prompt relay stuff, could also be done, but thats pretty much what "Extend Any Video" already do. Takes a video input and extends it

Yeah extend any video already does that but if hypothetically 😄 u made Prompt Relay with Video input (plus switch to toggle if it's video or images depending on what user wants or make it a separate workflow a part of the series) we'd get basically production pipeline where people could direct coherent longer scenes, from my tests this wf can already do continuous shots with great prompt following when length kept to 20 sec and length of segments properly adjusted and it can do much longer multi scene videos. IMO
Adding this functionality we'd get jack of all trades wf for story telling videos.

Sounds like a useful idea ;-) will give it a go.. Kijai also updated the node, so might come some updates to the "normal" wf as well.

Sounds like a useful idea ;-) will give it a go.. Kijai also updated the node, so might come some updates to the "normal" wf as well.

I did tests with LTX NAG Node today and it did not work with Prompt Relay Node it would crash it maybe update has fixed it, need to update nodes and check.
EDIT: updated comfy...nothing got broken in the process, NAG works now 🤤

Timeline Editor for Prompt Relay

Added new workflow to use Kijai's new timeline node, that makes it really intuitive and easy to adjust and add the sequences you want in your video.
Each sequence takes its own prompt for your story telling ;-) Drag and resize each sequence in the editor, and add as many sequences as you want/need

Can try it out here:
https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Movie-Maker

And if you want to try out the timeline workflow, you need to update the Prompt Relay node:
https://github.com/kijai/ComfyUI-PromptRelay

Requires updated ComfyUI-PromptRelay for the timeline node*
https://github.com/kijai/ComfyUI-PromptRelay

multi scene test, now need to test this new wf with Timeline editor 😀

wow ;-) that looks great ;-)
this node for sure has potential for some great story telling.. even a short little film with multiple generations (will add a continue from last video wf asap)

I'll make a variant of the timeline node, where its more wide AND can use input images as well..
So it can have a bit of best of both worlds ;-)

wow ;-) that looks great ;-)
this node for sure has potential for some great story telling.. even a short little film with multiple generations (will add a continue from last video wf asap)

yep, once u get good prompt following it's easy to focus on story and direction of video and get creative.

timeline WF is fine! 😀

nice nice ;-) siblings haha ... funny one ;-)

Timeline Editor for Prompt Relay + Multi reference Image

A variant of the timeline workflow where you can set reference images per segment, also available to try..

(renamed the older files, to be a little shorter.... the workflow are same though)

Requires updated ComfyUI-PromptRelay for the timeline node*
https://github.com/kijai/ComfyUI-PromptRelay

so far from

Timeline Editor for Prompt Relay + Multi reference Image

A variant of the timeline workflow where you can set reference images per segment, also available to try..

(renamed the older files, to be a little shorter.... the workflow are same though)

so far from my testing it does not work well, timeline wf with one image works well , already tested and posted a sample, but with this wf there is a messy generation, drifting away from prompted and hallucinating scenes , prompt adherence is all over the place, - it is like there is some antagonism between segments and images like they are fighting each other within model unlike normal story Prompt relay wf without timeline where it all easily connects harmoniously. I will post samples comparing results with both wfs a bit later...

EDIT: I think the problem is complex action and scenes with too long video so prompt following drifts away. I will do second run later today with length limit 20 sec.

Will test some here too, might be an issue with the new node. (or the new wf). Will check what it can be
I did see some odd results myself too, thought it was just me and the prompt. But if you had some too, maybe there is something with the new node (or new wf) that isnt quite right.
Will test ;-)

Will test some here too, might be an issue with the new node. (or the new wf). Will check what it can be
I did see some odd results myself too, thought it was just me and the prompt. But if you had some too, maybe there is something with the new node (or new wf) that isnt quite right.
Will test ;-)

This is with timeline plus relay node...took me about 8 hours to make it...I see preview and cancel maybe 20times it total to get this with 3 different sets of images because two other sets did not work...I had to reroll seed...

I plug in the same images and the same prompts in Relay story wf and first run it delivers what was prompted (480p so moving lips don't see well with this kind of shot)

so basically it seems timeline over-complicates already efficient workflow but Timeline works great with single image wf this is example ONLY one starting image and the same 4 prompts and the best result:

haha ;-) thats a creepy old lady

yes I ran some test with single image + timeline, and works great.
Not sure why multi-image + timeline would be any different, but will take a look ;-)

And yes, maybe with the multi-image workflow, not having timeline is plenty.

I love experimenting with this new timeline node... Attempting to use this for "extend any video" wf, but... meh.. not so easy. Guess the conditioning is overwriting the first part of the "overlap" in a way. So, the first segment in the timeline conditioning will start at idx 0, even tho it should be something like idx 100 (or whatever depending on overlap).... that means the transition is not smooth at all.
I have been able to make the animation seem smooth by creating a "first segment" the length of the overlap.. but ofc then the audio is completely messed up 😏

For sure very interesting. At it does work very well for I2V wf tho.

yes I also started on a continue video from last and generate more timeline segments from that. But as you also figured out, it wasnt as straight forwards as i hoped it would be ;-)
not sure if there is a way, but will see if i can trick it into working somehow

I did manage to get it working a little bit, but the output was a bit incoherent. But not sure if that was the prompt, not enough coffee or the wf ;-)

Updated Timeline Workflows*

The NAG node fight for the same crossattention as the prompt relay.
So thats removed in the updated Timeline workflows.

The updated timeline workflows should be more accurate, and if using with multi-images, the NAG node would mess up the timing, and images would inject different to the prompt segment (and potentially give all sorts of weirdness ;-))

(The prompt relay node has also been updated, if you update to latest KJNodes. You can now use seconds at the timeline instead of frames, if you prefer that)

It can be done.. extended 2 times using a modified extend video WF. Using ID-Lora + Prompt Relay.
What i found to work fairly well, was to generate the first video with the last segment about 1-2 seconds of "does not speak". Or else it has a slight tendensy to cut the audio in the extended video. Another tip is if you lets say have a 4 second "overlap" setting, and you prompt the extended video to speak something lasting 3-4 seconds, the first segment should be roughly 7-10 seconds to .. well.. without having other idea than "room for continuation without replacing"?

Using the ID-Lora, i use a "trim audio" node with the audio from the video capturing 4-5 seconds of fairly clean voice and use that as input. This way it does seem like it keeps the voice fairly consistent. I think with my hardware around 1 minute is max.. and as you see, some quality loss is happening at the end. It would probably be a lot better with proper scene changes i think.

PS. I had to do a 1/2 resolution on the video for upload, as it got a wee bit big.

Just want to add - Using the Prompt Relay node with ID-Lora does seem to work very well.. since the ID-Lora expects a particular setup with the [VISUAL]/[SPEECH]/[SOUND] setup, and that makes it really awkward to prompt for motions inbetween, but with the Prompt Relay, that works a lot better.

That video looks pretty good ;-) and some nice tips about the overlapping part.
And ID-Lora and Prompt Relay both use cross-attention, so they might crash a little, but if it worked maybe not too much

Had some test runs earlier with extending the video as well. Seems to work, so might upload a wf. Had a few very strange results too though, but think that comes down to expecting too much to happen in a short little segment, and if the model cant do all prompted for in that time, it seems to go a bit nuts haha making up stuff ..

Yeah, it can be a bit of a hit and miss sometimes. I do need to tweak the timings and do regeneration quite a few times, but i do feel its in the realm of "doable". Sometimes the ID-Lora does "crash" a bit, where the prompt makes the character say something.. well.. weird. I had a instance where i used [SPEECH]: "The weather is nice here" , and the model kinda said "The weather is nice woods", seemingly using some sort of logic solutions as to "here = woods" - As the scene IS in the woods. And you can hear a wee bit of misshap in the vide with the woods, where she repeats it, but i kept it as it could be something that WOULD happen in a RL situation of live recording :)

this updated wf without NAG works better, I got pretty much what I prompted with the same video I posted yesterday but for me it is not better than simple prompt relay with 4 segments and NAG on, also there is some bug with latest updates to KJ node and preview node causes OOM errors in all workflows that use it for example I cannot run Music video maker it gets oom at 4th or 5th segment or this updated wf in 720p it crashes due to OOM caused by Preview node here is the report I already asked chatgpt and it confirmed it is Preview Node that has issues, after disabling it I can run all workflows as before full 720p 2 minute videos with music video wf without crashing...note before timeline update it all worked with preview fine...

"torch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated : 5.50 GiB
Requested : 3.83 GiB
Device limit : 15.93 GiB
Free (according to CUDA): 0 bytes
PyTorch limit (set by user-supplied memory fraction)
: 17179869184.00 GiB
This error means you ran out of memory on your GPU.

TIPS: If the workflow worked before you might have accidentally set the batch_size to a large number.

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 534, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 334, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 308, in _async_map_node_over_list
await process_inputs(input_dict, i)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\execution.py", line 296, in process_inputs
result = f(**inputs)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy_api\internal_init_.py", line 149, in wrapped_func
return method(locked_class, **inputs)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy_api\latest_io.py", line 1826, in EXECUTE_NORMALIZED
to_return = cls.execute(*args, **kwargs)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy_extras\nodes_custom_sampler.py", line 963, in execute
samples = guider.sample(noise.generate_noise(latent), latent_image, sampler, sigmas, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise.seed)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 1052, in sample
output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\patcher_extension.py", line 113, in execute
return self.wrappers[self.idx](self, *args, **kwargs)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-kjnodes\nodes\ltxv_nodes.py", line 878, in call
out = executor(noise, latent_image, sampler, sigmas, denoise_mask, combined_callback, disable_pbar, seed, latent_shapes=latent_shapes)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\patcher_extension.py", line 105, in call
return new_executor.execute(*args, **kwargs)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 995, in outer_sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 981, in inner_sample
samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 751, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 1290, in sample_euler_cfg_pp
return sample_euler_ancestral_cfg_pp(model, x, sigmas, extra_args=extra_args, callback=callback, disable=disable, eta=0.0, s_noise=0.0, noise_sampler=None)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py", line 1267, in sample_euler_ancestral_cfg_pp
callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line 749, in
k_callback = lambda x: callback(x["i"], x["denoised"], x["x"], total_steps)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-kjnodes\nodes\ltxv_nodes.py", line 875, in combined_callback
new_callback(step, x0, x, total_steps)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-kjnodes\nodes\ltxv_nodes.py", line 842, in callback
preview_bytes = previewer.decode_latent_to_preview_image(preview_format, x0)

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-kjnodes\nodes\ltxv_nodes.py", line 625, in decode_latent_to_preview_image
num_images)).run()
~~~^^

File "threading.py", line 995, in run

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-kjnodes\nodes\ltxv_nodes.py", line 640, in process_previews
image_tensor = F.interpolate(image_tensor, (height,max_size), mode='nearest')

File "C:\ComfyUI_windows_portable_nvidia\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\functional.py", line 4817, in interpolate
return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
"

Thats good to know. Never had any issues with the sampler preview node myself.
But its optional just to have some previews at the sampler (you can still get previews at a lower quality, simply disconnect the vae from the preview node, maybe that works for you. Its super light, or should be).
I'll add a simple toggle for it, same way as there is for Prompt Enhance.

Since both are purely optional

The tiny VAE memory use is a problem, the VAE is currently storing the temporary results on GPU, this PR when merged should address it: https://github.com/Comfy-Org/ComfyUI/pull/13617

(VERY low resolution render, since i was just testing stuff... but you get the idea)

Adding some CFG might be a good choice for the more "complex" prompts with lots of things happening. Will upload a Dev model workflow for that. Still uses distilled lora, but more steps and cfg 3. It will be a little slower (compared to distilled only), but not a lot since its still with distilled lora. Set to 20 ish steps (but you can adjust the steps higher when needed)

Also noticed that VBVR reasoning lora for LTX can help with logic.. but that goes for any wf really. But since the prompt relay invites you to be a bit more creative, the VBVR did help me getting nice results a few times when the prompt was a bit "complex" (like opening doors, walking into this or that etc)

In case someone wanna try VBVR from https://huggingface.co/Video-Reason/ (but no comfy compatible yet at their repro... i think)
Comfy versions:
https://huggingface.co/siraxe/VBVR-LTX2.3-diffsynth_comfyui
https://huggingface.co/LiconStudio/Ltx2.3-VBVR-lora-I2V/

The tiny VAE memory use is a problem, the VAE is currently storing the temporary results on GPU, this PR when merged should address it: https://github.com/Comfy-Org/ComfyUI/pull/13617

This is currently merged into main comfy
For those who had some memory struggles try update Comfy

Prompt Relay with Custom Audio

Workflow added for prompt relay + custom audio. Single input image (but will add one for multi image as well).
And Kijai added an advanced option to the node to play around with for those who want to tinker and experiment ;-)

https://huggingface.co/RuneXX/LTX-2.3-Workflows/tree/main/Movie-Maker/Prompt-Relay-Custom-Audio

The tiny VAE memory use is a problem, the VAE is currently storing the temporary results on GPU, this PR when merged should address it: https://github.com/Comfy-Org/ComfyUI/pull/13617

This is currently merged into main comfy
For those who had some memory struggles try update Comfy

yeah it works fine now! 😃

Hello RuneXX.

Thank you as always for your helpful workflows.
Regarding LTX-2.3_-_I2V_T2V_Short-Story_PromptRelay-Timeline_custom_audio.json, even when I specify the width and height in the video settings, the generated video has a long side of 1536 pixels.

Could you please check this?

the generated video has a long side of 1536 pixels.
Could you please check this?

Ah... a little accident there, will fix that

Updated with correct resizing

the generated video has a long side of 1536 pixels.
Could you please check this?

Ah... a little accident there, will fix that

Updated with correct resizing

The new workflow is working correctly.
Thank you!

Prompt Relay + Custom Audio with multi reference image

Added a variant of the custom audio workflow that can take multi image references.
Although you can get quite the same with single reference image and prompting for "scene cut / new scene" etc with prompt describing the new scene, the multi ref image can give a bit more manual control.

A "music video" type of example, with song as custom audio.. but can be used for any type of storytelling :

Hello Rune, I was wondering if you had the multi image with Dev full model version of this workflow. Or would you suggest distilled 1.1 model version ?

Havent made one for the dev model.
Although usually all you need to do is choose the dev model at the model loader, and then under the manual sigma node in first pass, connect the Basic scheduler instead, and bump up the steps to 15-20 or so. And optionally also set the cfg to 2-3 or so.

But that is not fully using the dev potential, since it could benefit from a multi-modal guider etc.
Lately I have played around more with the dev model, with split sigmas, and other things to make it work better (the dev model can be a bit challenging sometimes).
Will try it out on this workflow, see if it works well

Yeah, well, there is no Basic Scheduler node in multi image promptrelay workflows, only LTXVscheduler. But in Custom Audio (Dev) multi image promptrelay version, there is both schedulers. Would it work if I bypass custom audio nodes so that it will generate the audio by itself ? By the way, prompt relay setup looks a bit different than latent-guider version, just saying :) Also take your time and thank you 👍