Spaces:
Running on Zero
Running on Zero
Fix joyfox runtime error: skip its AdaLN deltas
Browse filesThe Comfy-Org checkpoint joyfox was trained against gives adaln_proj an
8-dim input ([96768, 8]); the diffusers transformer projects the full
2688-dim time embedding, so those deltas cannot fold and raised a shape
error at switch time. Attention, MLP and both output heads still fold.
- README.md +4 -3
- h3_lora.py +6 -1
README.md
CHANGED
|
@@ -61,9 +61,10 @@ reference-tree layout as larry under a `diffusion_model.` prefix, so it folds th
|
|
| 61 |
`H3_REALISM=off` skips loading it.
|
| 62 |
|
| 63 |
A fourth set is [`joyfox/MiniMax-H3-Turbo`](https://huggingface.co/joyfox/MiniMax-H3-Turbo) (`joyfox`, 4 steps):
|
| 64 |
-
another ComfyUI-layout turbo LoRA, additionally covering the MLPs
|
| 65 |
-
|
| 66 |
-
|
|
|
|
| 67 |
|
| 68 |
## AoTI-compiled blocks
|
| 69 |
|
|
|
|
| 61 |
`H3_REALISM=off` skips loading it.
|
| 62 |
|
| 63 |
A fourth set is [`joyfox/MiniMax-H3-Turbo`](https://huggingface.co/joyfox/MiniMax-H3-Turbo) (`joyfox`, 4 steps):
|
| 64 |
+
another ComfyUI-layout turbo LoRA, additionally covering the MLPs and both output heads (`video_out` -> `proj_out`,
|
| 65 |
+
`audio_out` -> `audio_proj_out`), with a per-key `alpha` folded into `lora_B` at load. Its AdaLN deltas are skipped —
|
| 66 |
+
the Comfy-Org checkpoint uses an 8-dim modulation input where diffusers uses the full 2688-dim time embedding, so
|
| 67 |
+
they have no fold target. `H3_JOYFOX=off` skips loading it.
|
| 68 |
|
| 69 |
## AoTI-compiled blocks
|
| 70 |
|
h3_lora.py
CHANGED
|
@@ -29,7 +29,10 @@ The two supported LoRAs ship in different layouts:
|
|
| 29 |
* `joyfox` (`joyfox/MiniMax-H3-Turbo`) is another 4-step turbo LoRA, ComfyUI-native like realism but covering more
|
| 30 |
of the tree: attention, MLP, the block and final AdaLN projections, and both output heads (`final_layer.video_out`
|
| 31 |
-> `proj_out`, `final_layer.audio_out` -> `audio_proj_out`). Ranks are mixed (32 attention/MLP, 8 modulation/heads)
|
| 32 |
-
with a scalar `alpha` per key, so each `lora_B` is prescaled by `alpha / rank` at load (here always 1.0).
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
`H3_LORA` selects the larry file (`off` skips loading it), `H3_LIGHTX=off` skips lightx, `H3_REALISM=off` skips
|
| 35 |
realism, `H3_JOYFOX=off` skips joyfox, `H3_LORA_DEFAULT` picks which set starts folded, and `H3_LORA_STRENGTH` is
|
|
@@ -147,6 +150,8 @@ def _load_joyfox(inner_dim: int) -> dict:
|
|
| 147 |
bases = sorted({key.rsplit(".lora_", 1)[0].removeprefix("diffusion_model.") for key in lora if ".lora_" in key})
|
| 148 |
entries = []
|
| 149 |
for name in bases:
|
|
|
|
|
|
|
| 150 |
prefixed = f"diffusion_model.{name}"
|
| 151 |
a = lora[f"{prefixed}.lora_A.weight"]
|
| 152 |
# Per-key alpha (scalar tensor), unlike larry's fixed alpha == rank.
|
|
|
|
| 29 |
* `joyfox` (`joyfox/MiniMax-H3-Turbo`) is another 4-step turbo LoRA, ComfyUI-native like realism but covering more
|
| 30 |
of the tree: attention, MLP, the block and final AdaLN projections, and both output heads (`final_layer.video_out`
|
| 31 |
-> `proj_out`, `final_layer.audio_out` -> `audio_proj_out`). Ranks are mixed (32 attention/MLP, 8 modulation/heads)
|
| 32 |
+
with a scalar `alpha` per key, so each `lora_B` is prescaled by `alpha / rank` at load (here always 1.0). Its
|
| 33 |
+
`adaln_proj` entries are skipped: the Comfy-Org checkpoint it was trained against gives the modulation projection
|
| 34 |
+
an 8-dimensional input (`[96768, 8]`), while the diffusers transformer projects the full 2688-dim time embedding
|
| 35 |
+
(`[96768, 2688]`), so those deltas have no counterpart to fold into.
|
| 36 |
|
| 37 |
`H3_LORA` selects the larry file (`off` skips loading it), `H3_LIGHTX=off` skips lightx, `H3_REALISM=off` skips
|
| 38 |
realism, `H3_JOYFOX=off` skips joyfox, `H3_LORA_DEFAULT` picks which set starts folded, and `H3_LORA_STRENGTH` is
|
|
|
|
| 150 |
bases = sorted({key.rsplit(".lora_", 1)[0].removeprefix("diffusion_model.") for key in lora if ".lora_" in key})
|
| 151 |
entries = []
|
| 152 |
for name in bases:
|
| 153 |
+
if "adaln_proj" in name:
|
| 154 |
+
continue # 8-dim modulation input in the Comfy-Org tree vs 2688-dim in diffusers — cannot fold
|
| 155 |
prefixed = f"diffusion_model.{name}"
|
| 156 |
a = lora[f"{prefixed}.lora_A.weight"]
|
| 157 |
# Per-key alpha (scalar tensor), unlike larry's fixed alpha == rank.
|