# ClipProj v3

Four new matrices, calibrated against `qwen3vl_32b_minimax_h3_nvfp4_awq` — the
base MiniMax H3 encoder, the one a plain `Load CLIP` gives you.

Requires node **0.1.13** or later. The `-v3-mlp` files carry no linear matrix,
and an older node raises `KeyError: 'W'` on load.

## What changed

**Body descriptions.** While testing v2 we found that naming one part of a
body could rewrite the whole of it — build, height and face shifting together,
none of it asked for. Those matrices were calibrated against a modified 32B.
v3 is calibrated against the stock encoder, and we no longer observe the
problem: a build and an attribute stated in the same prompt are honoured
independently.

**Everything is measurably closer to the 32B.** Measured on a single reference
prompt, every projection encoded against the same stock 32B, so the numbers are
comparable to each other — which the figures published with v1 and v2 were not,
having been measured against different targets and in different spaces. The
`cos_test` written inside each file is the training-time figure against that
campaign's own target and does **not** compare across versions; the comparable
number is stored separately as `cos_prompt_reference`.

The render pipeline was verified deterministic before measuring anything: the
same workflow run twice, thirteen days apart, produced two files with identical
SHA-256. Any difference below therefore comes from the projection and nothing
else.

| projection | mean cosine vs 32B |
|---|---|
| 8B v3-mlp | 0.9449 |
| 8B v2 `celeb-mlp` | 0.9393 |
| 4B v3-mlp | 0.9381 |
| 4B v2 `celeb-mlp` | 0.9293 |
| 8B v3 | 0.9289 |
| 4B v3 | 0.9193 |

**The 8B now sees image tokens.** Until v3 the image corpus had only ever been
encoded with a 4B student, so both 8B matrices projected vision tokens without
having seen a single one — while the node accepts a reference image. Measured on
100 held-out images, on the raw conditioning the diffusion model actually
receives:

| | vision tokens | text in the same sequences |
|---|---|---|
| 8B residual, before | 0.7692 | 0.9085 |
| 8B residual, after | 0.8578 | 0.9605 |
| 8B matrix, before | 0.7845 | 0.8926 |
| 8B matrix, after | 0.8457 | 0.9361 |

That costs 0.0027 of pure-text cosine on the residual and 0.0013 on the matrix —
which is why the 8B figures above are slightly below what a text-only corpus
would have given.

**Prefer the `-mlp` files on the measurement, not on this scene.** They sit
closer to the 32B, 0.9449 against 0.9289 on the 8B. But on this prompt the five
renders are faithful, plain matrices included: the pose, the dress, the white
pieces, the cat, the straw hat, the laundry, the bouncing knee all hold on all
five. A tightly written prompt survives even the linear baseline, and the
difference between the files shows up in the numbers well before it shows up on
screen.

## What this does not fix, and will not

A projection cannot invent information the small encoder never had. Five hours
of training on a 3090, two more to encode the dataset, six and a half million
tokens — none of that changes what a 4B model wrote down in the first place.
The result is an approximation of the 32B's conditioning, not a copy of it.

In practice the line falls here: **what the prompt states, the projection
carries; what the prompt leaves open, the model fills from its own prior.**
Constrain a scene tightly and the projected renders track the 32B closely.
Leave the set dressing unstated — a cat somewhere, laundry on a line, furniture
— and it will be furnished differently. That is not a defect to be tuned away,
it is what a 20-degree angle between two conditioning vectors looks like on
screen.

## Training data

The aim was to activate as much of the encoder's weight space as possible
rather than to cover one domain deeply. A matrix only learns to project the
directions it has actually seen used, so the corpus deliberately mixes
registers, languages and lengths.

| source | tokens |
|---|---|
| cinematic video prompts | 1 342 987 |
| native H3 format, 4 length draws | 3 169 879 |
| explicit register | 544 073 |
| Chinese | 532 302 |
| celebrity prompts, long form | 314 516 |
| filler sequences | 149 917 |
| celebrity prompts, short form | 99 668 |
| images, 3 blocks, 1 700 images | 349 244 |
| **total, both students** | **6 502 586** |

Both students now see the same corpus, images included — which makes the 4B and
the 8B comparable to each other for the first time.

3 331 prompts for fitting, one in fifty held out for measurement. Tap 24 on
both students. Sequence lengths are drawn at random rather than truncated to a
fixed size, so a given word appears at many different positions instead of
always the same ones.

## The demo folder

`demo/` holds five renders of the same scene. Same prompt, same seed 42, same
8 steps, same turbo LoRA, same DiT, same VAE — only the projection changes. The
pipeline is reproducible bit for bit, verified by running it twice and comparing
the decoded video and audio streams, so every difference between these five
comes from the projection and nothing else.

| file | conditioning |
|---|---|
| `chess-32b-reference.mp4` | Qwen3-VL-32B, 15.7 GB |
| `chess-8b-mlp.mp4` | Qwen3-VL-8B + `v3-mlp`, 10.0 GB |
| `chess-4b-mlp.mp4` | Qwen3-VL-4B + `v3-mlp`, 4.8 GB |
| `chess-8b-ridge.mp4` | Qwen3-VL-8B + `v3`, matrix only |
| `chess-4b-ridge.mp4` | Qwen3-VL-4B + `v3`, matrix only |
| `chess-comparison.mp4` | the five in sequence, labelled |
| `chess-prompt.txt` | the prompt, verbatim |

The comparison runs matrix-only first, then the 32B, then the residuals — so the
reference sits in the middle and each half is read against it.

**You will not reproduce these files byte for byte, and that is normal.** Noticed
while testing something else, so take it as an observation rather than a study:
the result depends on the model of GPU the encoder runs on. Four cards, one
prompt, one seed, four different outputs — while two different RTX 3090s gave
byte-identical video. Not the architecture either: the 3060 and the 3090 are
both Ampere and disagree. Encoding the same prompt on two cards gives
conditioning that matches to a relative error of 7 × 10⁻⁷; eight denoising steps
turn that into a different piece of furniture. On one machine everything here is
reproducible to the bit, which is what makes the five-way comparison meaningful.

Watch her knee. The prompt asks for it three times, ending on a sentence of its
own: *"Her knee never stops bouncing."* It is the most redundant instruction in
the text, it is a continuous involuntary motion with no narrative purpose, and
it is the clearest single indicator that a projection carried what was written.
Then watch the cat, the laundry and the furniture, which are named once and
anchored nowhere — those move, and they are supposed to.

One thing none of the five gets right, including the 32B: she lifts a knight and
does not put it back on the same square. Object permanence through an occluding
hand on a grid of sixty-four identical squares is a limit of the video model, not
of the conditioning. It is listed here so nobody attributes it to the projection.

## Files

| file | size | structure | encoder |
|---|---|---|---|
| `mmh3-4b-ClipProj-v3-mlp.safetensors` | 503 MB | residual only | Qwen3-VL-4B |
| `mmh3-8b-ClipProj-v3-mlp.safetensors` | 604 MB | residual only | Qwen3-VL-8B |
| `mmh3-4b-ClipProj-v3.safetensors` | 26 MB | matrix only | Qwen3-VL-4B |
| `mmh3-8b-ClipProj-v3.safetensors` | 42 MB | matrix only | Qwen3-VL-8B |

The residuals use a hidden width of 32 768 against 16 384 in v2, which is where
the extra download size comes from.