img2vid fp16 fix
#23
by loretoparisi - opened
This fix can be applied to https://huggingface.co/stabilityai/stable-video-diffusion-img2vid?
This could necessary when using force_upcast:
force_upcast (`bool`, *optional*, default to `True`):
If enabled it will force the VAE to run in float32 for high image resolution pipelines, such as SD-XL. VAE
can be fine-tuned / trained to a lower range without loosing too much precision in which case
`force_upcast` can be set to `False` - see: https://huggingface.co/madebyollin/sdxl-vae-fp16-fix
No. sdxl-vae-fp16-fix is a replacement for the broken-in-fp16 SDXL VAE https://huggingface.co/stabilityai/sdxl-vae, but https://huggingface.co/stabilityai/stable-video-diffusion-img2vid uses a different, temporal VAE which AFAIK isn't broken in fp16. So unless you're seeing solid-black/NaN videos, you can probably just set force_upcast=False with the existing SVD VAE and use it in fp16.