akhaliq HF Staff commited on
Commit
e2db4ca
·
1 Parent(s): 9ceee9b

Fix joyfox runtime error: skip its AdaLN deltas

Browse files

The Comfy-Org checkpoint joyfox was trained against gives adaln_proj an
8-dim input ([96768, 8]); the diffusers transformer projects the full
2688-dim time embedding, so those deltas cannot fold and raised a shape
error at switch time. Attention, MLP and both output heads still fold.

Files changed (2) hide show
  1. README.md +4 -3
  2. h3_lora.py +6 -1
README.md CHANGED
@@ -61,9 +61,10 @@ reference-tree layout as larry under a `diffusion_model.` prefix, so it folds th
61
  `H3_REALISM=off` skips loading it.
62
 
63
  A fourth set is [`joyfox/MiniMax-H3-Turbo`](https://huggingface.co/joyfox/MiniMax-H3-Turbo) (`joyfox`, 4 steps):
64
- another ComfyUI-layout turbo LoRA, additionally covering the MLPs, the AdaLN projections, and both output heads
65
- (`video_out` -> `proj_out`, `audio_out` -> `audio_proj_out`), with a per-key `alpha` folded into `lora_B` at load.
66
- `H3_JOYFOX=off` skips loading it.
 
67
 
68
  ## AoTI-compiled blocks
69
 
 
61
  `H3_REALISM=off` skips loading it.
62
 
63
  A fourth set is [`joyfox/MiniMax-H3-Turbo`](https://huggingface.co/joyfox/MiniMax-H3-Turbo) (`joyfox`, 4 steps):
64
+ another ComfyUI-layout turbo LoRA, additionally covering the MLPs and both output heads (`video_out` -> `proj_out`,
65
+ `audio_out` -> `audio_proj_out`), with a per-key `alpha` folded into `lora_B` at load. Its AdaLN deltas are skipped —
66
+ the Comfy-Org checkpoint uses an 8-dim modulation input where diffusers uses the full 2688-dim time embedding, so
67
+ they have no fold target. `H3_JOYFOX=off` skips loading it.
68
 
69
  ## AoTI-compiled blocks
70
 
h3_lora.py CHANGED
@@ -29,7 +29,10 @@ The two supported LoRAs ship in different layouts:
29
  * `joyfox` (`joyfox/MiniMax-H3-Turbo`) is another 4-step turbo LoRA, ComfyUI-native like realism but covering more
30
  of the tree: attention, MLP, the block and final AdaLN projections, and both output heads (`final_layer.video_out`
31
  -> `proj_out`, `final_layer.audio_out` -> `audio_proj_out`). Ranks are mixed (32 attention/MLP, 8 modulation/heads)
32
- with a scalar `alpha` per key, so each `lora_B` is prescaled by `alpha / rank` at load (here always 1.0).
 
 
 
33
 
34
  `H3_LORA` selects the larry file (`off` skips loading it), `H3_LIGHTX=off` skips lightx, `H3_REALISM=off` skips
35
  realism, `H3_JOYFOX=off` skips joyfox, `H3_LORA_DEFAULT` picks which set starts folded, and `H3_LORA_STRENGTH` is
@@ -147,6 +150,8 @@ def _load_joyfox(inner_dim: int) -> dict:
147
  bases = sorted({key.rsplit(".lora_", 1)[0].removeprefix("diffusion_model.") for key in lora if ".lora_" in key})
148
  entries = []
149
  for name in bases:
 
 
150
  prefixed = f"diffusion_model.{name}"
151
  a = lora[f"{prefixed}.lora_A.weight"]
152
  # Per-key alpha (scalar tensor), unlike larry's fixed alpha == rank.
 
29
  * `joyfox` (`joyfox/MiniMax-H3-Turbo`) is another 4-step turbo LoRA, ComfyUI-native like realism but covering more
30
  of the tree: attention, MLP, the block and final AdaLN projections, and both output heads (`final_layer.video_out`
31
  -> `proj_out`, `final_layer.audio_out` -> `audio_proj_out`). Ranks are mixed (32 attention/MLP, 8 modulation/heads)
32
+ with a scalar `alpha` per key, so each `lora_B` is prescaled by `alpha / rank` at load (here always 1.0). Its
33
+ `adaln_proj` entries are skipped: the Comfy-Org checkpoint it was trained against gives the modulation projection
34
+ an 8-dimensional input (`[96768, 8]`), while the diffusers transformer projects the full 2688-dim time embedding
35
+ (`[96768, 2688]`), so those deltas have no counterpart to fold into.
36
 
37
  `H3_LORA` selects the larry file (`off` skips loading it), `H3_LIGHTX=off` skips lightx, `H3_REALISM=off` skips
38
  realism, `H3_JOYFOX=off` skips joyfox, `H3_LORA_DEFAULT` picks which set starts folded, and `H3_LORA_STRENGTH` is
 
150
  bases = sorted({key.rsplit(".lora_", 1)[0].removeprefix("diffusion_model.") for key in lora if ".lora_" in key})
151
  entries = []
152
  for name in bases:
153
+ if "adaln_proj" in name:
154
+ continue # 8-dim modulation input in the Comfy-Org tree vs 2688-dim in diffusers — cannot fold
155
  prefixed = f"diffusion_model.{name}"
156
  a = lora[f"{prefixed}.lora_A.weight"]
157
  # Per-key alpha (scalar tensor), unlike larry's fixed alpha == rank.