README for 0.1.3: named people, the -mlp compatibility trap, corpus sources

README.md

@@ -12,100 +12,117 @@ base_model:
 library_name: comfyui
 ---
 
-# ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B
+# ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B or 8B
 
-**Projection matrices that let a Qwen3-VL-4B replace the Qwen3-VL-32B text encoder of MiniMax H3.**
+**Projection matrices that let a small Qwen3-VL replace the Qwen3-VL-32B text encoder of MiniMax H3.**
 
-**15.7 GB → 5.2 GB of VRAM**, with no change to the diffusion model, the VAEs or the sampler.
+**15.7 GB → 4.5 GB of VRAM**, with no change to the diffusion model, the VAEs or the sampler.
 
 > ⚠️ **Proof of concept — working, but a proof of concept.** It runs and produces good video, and every number below was measured on real hardware. Built and tested on a single setup (Windows 11, NVIDIA, ComfyUI 0.31.0) with deliberately limited exploration.
 
 These files are useless on their own. They require the custom node:
 **[github.com/nicolab28/ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)**
 
-## Now in safetensors
+## Update to node 0.1.3, and re-download
 
-Every matrix is available as `.safetensors`. **Prefer it.** A `.pt` goes through pickle, which can execute arbitrary code the moment it is opened — an absurd risk for files that hold nothing but tensors and a handful of scalars. The scalars now live in the safetensors header and the node converts them back on load.
+Two reasons, one of them silent.
 
-Reading them needs node **v0.1.2**. The `.pt` files are kept alongside for now so that anyone still on an older node has something that works; they will go away in a later release.
+**The `-mlp` matrices carry a residual network, and an older node ignores it without saying so.** It reads the matrix, finds keys it does not know, drops them, and applies the linear part alone. Nothing fails, nothing warns, and you end up judging the plain matrix while believing you tested the residual. Node 0.1.3 reads them.
 
-*Requested by @lucas-ai26, who was right.*
-
-## Re-download if you got the first release
-
-**The first release was missing the attention-sink vector, and short prompts suffered for it.** The first token of a sequence is an attention sink: its direction is constant from one prompt to the next (cosine 1.0000 measured over 1966 prompts) and it carries nothing from the text, yet its norm reaches 16 500 against 291 for a text token. Calibration excluded it, rightly — its extreme values would wreck the statistics — but the node projected it anyway, through a matrix that had never seen one, producing an arbitrary vector of enormous norm. Invisible on a 200-token prompt where it is 0.5 % of the positions, ruinous on a 7-token one where it is 14 %.
-
-Every matrix here now carries the measured sink vector and substitutes it at inference. Since the vector is constant, this is not an approximation: it is the value the 32B would have produced.
-
-Also new in this release: an **8B matrix**, an **int8_convrot matrix**, and **ref2va now works** — see below.
+**Everything is renamed.** The old `h3_qwen3vl_*` files have moved to `obsolete/` and the `.pt` copies are gone: opening a pickle executes code, which makes no sense for a file holding six tensors. If a workflow of yours names an old file, either point it at `obsolete/` or, better, switch to the new set.
 
 ## What this is
 
-MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a `[seq, 5120]` tensor. This repository provides a learned linear map that lets a much smaller Qwen3-VL produce the same conditioning:
+MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a `[seq, 5120]` tensor. This repository provides a learned map that lets a much smaller Qwen3-VL produce the same conditioning:
 
 ```
 cond = ((h - mean_in) / std_in) @ W * std_out + mean_out
 ```
 
-It works because every Qwen3-VL shares the **same tokenizer** (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. Calibration is plain **ridge regression** — no gradients, no epochs, no learning rate.
+and, in the `-mlp` files, plus the output of a small residual network fed the same standardised input.
+
+It works because every Qwen3-VL shares the **same tokenizer** (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. The matrix is fitted by plain **ridge regression** — no gradients, no epochs, no learning rate. The residual network is the only part that is trained.
 
 ## Files
 
 Put them in `ComfyUI/models/clip_projections/`.
 
-**Start with `h3_qwen3vl_4b_tap24`.** It is calibrated on eight times more data than anything else here.
+**Start with `mmh3-8b-ClipProj-celeb-mlp` if you have the VRAM, `mmh3-4b-ClipProj-celeb-mlp` otherwise.**
 
-| File | Use it for | Encoder | Corpus | Test cosine |
+| File | Encoder | Names covered | Residual | Test cosine |
 |---|---|---|---|---|
-| `h3_qwen3vl_4b_tap24` | **the default** | any Qwen3-VL-4B | 2 000 prompts | 0.712 |
-| `h3_qwen3vl_8b_tap24` | any 8B encoder | any Qwen3-VL-8B | 203 prompts | 0.731 |
-| `h3_control_identity` | control — run it once | — | — | — |
-| `h3_control_zero` | control — run it once | — | — | — |
-| `h3_qwen3vl_4b_CONDPROJ_tap24` | see below | any Qwen3-VL-4B | 240 prompts | 0.845 * |
-| `h3_qwen3vl_8b_CONDPROJ_tap24` | see below | any Qwen3-VL-8B | 203 prompts | 0.860 * |
-| `h3_qwen3vl_4b_int8convrot_tap24` | comparison only | any Qwen3-VL-4B | 240 prompts | 0.697 |
-| `h3_qwen3vl_4b_tap24_200prompts` | comparison only | any Qwen3-VL-4B | 200 prompts | 0.699 |
+| `mmh3-4b-ClipProj` | any Qwen3-VL-4B | no | no | 0.7169 |
+| `mmh3-4b-ClipProj-mlp` | any Qwen3-VL-4B | no | yes | 0.7944 |
+| `mmh3-4b-ClipProj-celeb` | any Qwen3-VL-4B | **yes** | no | 0.7095 |
+| `mmh3-4b-ClipProj-celeb-mlp` | any Qwen3-VL-4B | **yes** | yes | 0.7930 |
+| `mmh3-8b-ClipProj` | any Qwen3-VL-8B | no | no | 0.7528 |
+| `mmh3-8b-ClipProj-mlp` | any Qwen3-VL-8B | no | yes | 0.7970 |
+| `mmh3-8b-ClipProj-celeb` | any Qwen3-VL-8B | **yes** | no | 0.7466 |
+| `mmh3-8b-ClipProj-celeb-mlp` | any Qwen3-VL-8B | **yes** | yes | **0.8037** |
+| `mmh3-ClipProj-control-zero` | — | control, run it once | — | — |
+| `mmh3-ClipProj-control-identity` | — | control, run it once | — | — |
 
-\* measured in a different space and **not comparable to the other rows**. See below, and do not read it as better.
+All eight are calibrated on the same general corpus and measured on the same held-out prompts, so the column is comparable across every row.
 
-Every matrix works on **any variant of its own size**: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on. The 8B matrices need an 8B encoder though, 4096 input dimensions instead of 2560, and the node checks and refuses a mismatch.
+Every matrix works on **any variant of its own size**: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on. The 8B matrices need an 8B encoder though — 4096 input dimensions instead of 2560 — and the node checks the width and refuses a mismatch.
 
-The **8B matrices need an 8B encoder** — different input width, 4096 instead of 2560, so they are not interchangeable with the 4B ones. The node checks the width and refuses a mismatch.
+## Named people
 
-Every matrix here works on **any variant of its own size**: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on.
+**This is what changed in 0.1.3, and it was a corpus problem.**
 
-## The CONDPROJ matrices, and why they are not the default
+The calibration corpus named a person on about 70 lines out of 8632, roughly 0.02 % of the training tokens. The directions of the hidden space that carry an identity were therefore constrained by nothing at all, and the fit put whatever minimised the error on landscape descriptions there. Named people came out as somebody else.
 
-They are kept because the comparison posted on r/StableDiffusion ran on them, so the links have to keep working. They are **not better**, and the story is worth telling because the mistake was instructive.
+The `-celeb` matrices add 500 people, ranked by popularity, with five short prompts and two long ones each. What it buys and what it costs:
 
-The DiT does not consume the conditioning as it arrives: it first passes it through `condition_proj`, a `Linear(5120 → 5376)` feeding the token refiner. That layer's spectrum is very uneven, a factor of 45 between the top and bottom deciles of its singular values, and 52 % of the energy in 10 % of the directions. Plain ridge regression ignores this and spends as much effort on a direction the DiT will multiply by 0.10 as on one it will multiply by 37. Calibrating against the **output** of that layer instead, then mapping back through the pseudo-inverse, should therefore minimise the error the DiT actually sees. The cosine went from 0.697 to 0.845 on the 4B and 0.731 to 0.860 on the 8B.
+| | name tokens | rest of the sentence | general test set |
+|---|---|---|---|
+| without | 0.8265 | 0.9358 | 0.7944 |
+| with | **0.8844** | **0.9516** | 0.7930 |
 
-Then I compared what the two matrices actually output, on random inputs:
+Seven thousandths of cosine on the general corpus, for six points on the tokens that carry an identity. The rest of the sentence improves too, because the celebrity prompts are short and the general corpus had nothing under fifteen words.
 
-```
-4B   CONDPROJ against unweighted, same corpus     cosine 0.999998
-8B   CONDPROJ against unweighted, same corpus     cosine 0.999999
-4B   CONDPROJ against the 2000-prompt matrix      cosine 0.744034
-```
+Two findings that decide how far this is worth pushing.
 
-They are the same function. The entire gain was an artefact of measuring in a different space: applying an invertible transform to both vectors before taking a cosine flatters agreement on the dominant directions.
+**Two contexts per person are enough.** Measured on contexts held out for people the matrix had seen: 0.9875 at two, 0.9945 at five, 0.9986 at twenty. Forty is a waste.
 
-The reason is simple in hindsight. Unregularised least squares is invariant to an invertible linear transform of the targets, so fitting in one space and mapping back recovers the same map. Only the ridge penalty breaks that invariance, and with 37 851 training tokens against λ = 1000 it barely binds. The idea would matter with far less data or far stronger regularisation. It does not matter here.
+**Five hundred names generalise to names never seen.** A held-out band at popularity ranks 501 to 540, absent from every calibration, reconstructs at 0.8795 against 0.8844 for the covered ones. Covering 500 people does not teach 500 names; it teaches the map how to handle that region of the space. Going to several thousand would buy very little.
 
-*The idea came from u/stddealer on r/StableDiffusion, and it was a good one. The measurement is on me: I published the cosine before checking whether the matrix had changed at all.*
+**What still fails is not the corpus.** Characters whose identity is a mask rather than a face come out as a stranger wearing the right costume. People whose fame predates the era when everything was photographed come out wrong or generic. And some names fail on the plain 32B too, so run the reference before blaming the projection — that check has overturned three of my own conclusions.
+
+## Where the calibration data comes from
+
+The general corpus is [GokuScraper/seedance-2-prompts-datasets](https://huggingface.co/datasets/GokuScraper/seedance-2-prompts-datasets), filtered to prompts of fifteen words or more and deduplicated: 8632 lines, median 128 words. The 500 named people come from a TMDB export published on Kaggle, ranked by popularity, with transliterated names dropped beyond rank 1000.
+
+Around each name, five short prompts are generated from templates, and two longer ones in MiniMax H3's section format are written by Mistral Small and Gemini Flash Lite, half each. Everything needed to rebuild the corpus is in the node's `calibration/` folder, including the system prompt the long prompts were written from.
+
+## The residual network
+
+The `-mlp` files carry a `d_in → 16384 → 5120` network with a GELU, added to the matrix rather than replacing it. Its last layer is initialised to zero, so at the first step the model reproduces the matrix exactly and can only improve on it. It is worth 0.05 to 0.08 of cosine, four times what multiplying the corpus by eleven buys the linear map.
+
+**Which of the two renders better is not settled.** The cosine does not predict it — that is the single most repeated lesson of this project. Try both on your own prompts.
+
+Two things measured while building it. Width beats depth: at equal parameter count, two hidden layers of 8192 reach 0.7691 against 0.7944 for one layer of 16384. And a residual extrapolates worse than a matrix does — outside the corpus it saw, a linear map degrades gracefully while the network collapses.
 
 ## Measured results
 
-| Corpus | Tokens | Cross-prompt CKA | Test cosine | Test R² |
-|---|---|---|---|---|
-| 200 prompts | 37 361 | 0.95 | 0.699 | 0.490 |
-| 2 000 prompts | 288 608 | 0.92 | **0.712** | 0.507 |
+| | 4B | 8B |
+|---|---|---|
+| matrix, no names | 0.7169 | 0.7528 |
+| matrix + residual | 0.7944 | 0.7970 |
+| matrix, names covered | 0.7095 | 0.7466 |
+| matrix + residual, names covered | 0.7930 | **0.8037** |
+
+A cosine of 0.79 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts with several distinct cuts and no bleed between them, fl2va with first and last frame, ref2va with a reference image, and since 0.1.3 ref2va with a reference video.
+
+Fidelity does **not** collapse on short prompts: measured per-token cosine goes from 0.937 at 80 words to 0.908 at 2 words, once the attention sink is handled.
 
-Eight times the data buys 1.8 % of cosine: the linear projection is **at its ceiling**, not starved of data. Going further needs an MLP, not more prompts.
+## Speech
 
-A cosine of 0.71 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts (four distinct cuts with no bleed between them), fl2va with first and last frame, and ref2va with a reference image.
+The first release lost non-English speech: a French line came out half Spanish, and the 8B put everything in English. That was the clearest regression and I could not explain it then.
 
-Fidelity does **not** collapse on short prompts, contrary to what one might expect from a corpus of long ones: measured per-token cosine goes from 0.937 at 80 words down to 0.908 at 2 words, a 3 % decline, once the sink is handled.
+With `mmh3-8b-ClipProj-celeb-mlp`, a three-shot clip carrying English, French and Spanish comes out like the 32B does, and the audio level gap measured against the reference has gone from 7.6 dB to 3.5.
+
+Part of what was blamed on the projection was not the projection. A line that fills more than about two thirds of its shot comes out slurred whatever encoder produced the conditioning — the fix is a longer shot, not a better matrix. And MiniMax H3 expects speech wrapped in `<d>[Language] ...</d>` with a stable speaker id declared beforehand; without that, one voice with one accent is used for the whole clip. Neither of those is documented here because neither is ours, but both cost me a day.
 
 ## Run the controls first
 
@@ -113,25 +130,40 @@ The two control matrices exist to prove the learned matrix is doing the work rat
 
 | Matrix | Output for *"a red ball on a wood table"* |
 |---|---|
-| `h3_control_zero.safetensors` | a countryside landscape — the prompt is entirely ignored |
-| `h3_control_identity.safetensors` | a golden object in flames — unusable |
-| `h3_qwen3vl_4b_tap24.safetensors` | the red ball on a wood table |
+| `mmh3-ClipProj-control-zero` | a countryside landscape — the prompt is entirely ignored |
+| `mmh3-ClipProj-control-identity` | a golden object in flames — unusable |
+| a learned matrix | the red ball on a wood table |
 
 `‖W_identity‖ = 50.6` against `‖W_learned‖ = 52.4` — near-identical energy, so the difference is structural, not a matter of scale.
 
 **If the identity control ever looks fine, the learned matrix adds nothing — and you want to know that before trusting it.**
 
-## Known limitations
+## What is in obsolete/
 
-**Named references depend on the encoder, not on the projection.** This was measured rather than guessed: asking the encoder to describe a person in plain text — which bypasses the matrix entirely — shows where the knowledge actually stops. The 4B places Scarlett Johansson correctly as Black Widow but believes she has dark brown hair; the 8B describes her correctly as blonde with blue eyes. So when a proper noun renders as the wrong person, question the encoder first.
+The previous matrices, kept because a comparison posted on r/StableDiffusion ran on them and the links have to keep working. They have no name coverage and are calibrated on a corpus thirty times smaller. There is no reason to prefer them.
 
-The practical workaround is to describe rather than name: *"the actress X as [role], blonde, ..."* recovers an identity that the bare name loses, on both the 4B and the 8B. A name is a fragile signal carried by two or three tokens; a description spreads it over a dozen redundant ones and the reconstruction error averages out instead of accumulating.
+Among them, the `CONDPROJ` pair, and the story is worth telling because the mistake was instructive.
+
+The DiT does not consume the conditioning as it arrives: it first passes it through `condition_proj`, a `Linear(5120 → 5376)` feeding the token refiner. That layer's spectrum is very uneven — a factor of 45 between the top and bottom deciles of its singular values, 52 % of the energy in 10 % of the directions. Plain ridge regression ignores this and spends as much effort on a direction the DiT will multiply by 0.10 as on one it will multiply by 37. Calibrating against the **output** of that layer instead, then mapping back through the pseudo-inverse, should therefore minimise the error the DiT actually sees. The cosine went from 0.697 to 0.845 on the 4B and 0.731 to 0.860 on the 8B.
+
+Then I compared what the two matrices actually output:
+
+```
+4B   CONDPROJ against unweighted, same corpus     cosine 0.999998
+8B   CONDPROJ against unweighted, same corpus     cosine 0.999999
+```
+
+They are the same function. Unregularised least squares is invariant to an invertible linear transform of the targets, so fitting in one space and mapping back recovers the same map; only the ridge penalty breaks that invariance, and with 37 851 training tokens against λ = 1000 it barely binds. The entire gain was an artefact of measuring in a different space.
+
+*The idea came from u/stddealer on r/StableDiffusion, and it was a good one. The measurement is on me: I published the cosine before checking whether the matrix had changed at all.*
+
+## Known limitations
 
-Note that this does not apply to **ref2va**: there the identity comes from the reference image, so the encoder's factual gaps stop mattering.
+**Quantisation costs facts.** Comparing `int8_convrot` against `bf16` on factual recall shows errors appearing under quantisation. Fine for general use, worth knowing if your prompts lean on proper nouns.
 
-**Quantisation costs facts.** The same test on `int8_convrot` against `bf16` shows two factual errors appearing under quantisation. Fine for general use, worth knowing if your prompts lean on proper nouns.
+**Masks defeat identity.** A character recognised by a costume rather than a face comes out as an unknown person in the right suit. No corpus fixes that, because the identity is not in the name's representation to begin with.
 
-**Speech in languages other than English degrades.** Reported and reproduced with French: the 32B pronounces it cleanly, a projected 4B or 8B does not. This is *not* a corpus problem — measured on identical English prompts differing only in the quoted line, French tokens reconstruct at 0.8974 against 0.8996 for English, which is noise. A cosine of 0.90 is ample for visual semantics and insufficient for phonetics; the DiT's audio branch is far more demanding than its image branch, and a language the model handles less confidently has less margin to absorb the error.
+**Counting is unreliable, and not because of the projection.** Ask for three of something and you get four, on the 32B too. Enumerating works better than announcing a number.
 
 ## Required models
 
@@ -139,7 +171,7 @@ Note that this does not apply to **ref2va**: there the identity comes from the r
 |---|---|
 | Diffusion model + VAEs | [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) |
 | Text encoder, 4B | [Comfy-Org/Krea-2](https://huggingface.co/Comfy-Org/Krea-2) → `text_encoders/qwen3vl_4b_fp8_scaled.safetensors` |
-| Text encoder, 8B | any ComfyUI-format Qwen3-VL-8B (the 8B matrix expects 4096 input dims) |
+| Text encoder, 8B | any ComfyUI-format Qwen3-VL-8B (the 8B matrices expect 4096 input dims) |
 
 The 32B text encoder is **no longer needed** — that is the entire point.
 
@@ -158,4 +190,4 @@ You remain responsible for what you generate and for complying with the licences
 
 ## Credits
 
-Vibe-coded with **Anthropic Claude Code (Opus 5)**. Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected.
+Vibe-coded with **Anthropic Claude Code (Opus 5)**. Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected. Three claims in the previous version of this file were wrong and are corrected here.