how did you settle on layer 16 of the 8B and layer 29 of the 32B?

Sorry for the long response; there was a lot here. Because the process of picking that layer is what led to my previous comment about bimodality of images.

Main Test

In a nutshell, for every input token you get 1x [float;4096] hidden-state vector from the 8B and 1x [float;5120] hidden-state vector from the 32B. So for some input producing N tokens, at any particular layer you can treat the hidden states as an N × 4096 matrix for 8B and an N × 5120 matrix for 32B.

You can then compute linear Centered Kernel Alignment between those matrices as a rough measure of how similarly the two models are representing the same set of token positions. CKA is insensitive to things like rotation and global scaling, so a high value makes that layer pair an interesting candidate for a learned linear/affine bridge.

N is variable here. I never forced every image to have the same token count; it was simply however many tokens that image produced.

The layer hunt is basically exhaustively testing these combination pairs:

$$cka(H_1^{32b},H_1^{8b})cka(H\_\{1\}^\{32b\},\; H\_\{1\}^\{8b\})$$

$$cka(H_1^{32b},H_2^{8b})cka(H\_\{1\}^\{32b\},\; H\_\{2\}^\{8b\})$$

$$cka(H_1^{32b},H_3^{8b})cka(H\_\{1\}^\{32b\},\; H\_\{3\}^\{8b\})$$

$$\mathrm{.}\mathrm{.}\mathrm{.}...$$

$$cka(H_2^{32b},H_2^{8b})cka(H\_\{2\}^\{32b\},\; H\_\{2\}^\{8b\})$$

$$cka(H_2^{32b},H_3^{8b})cka(H\_\{2\}^\{32b\},\; H\_\{3\}^\{8b\})$$

$$\mathrm{.}\mathrm{.}\mathrm{.}...$$

$$cka(H_n^{32b},H_m^{8b})cka(H\_\{n\}^\{32b\},\; H\_\{m\}^\{8b\})$$

Notation

• cka: linear Centered Kernel Alignment
• H: Hidden-state matrix.
• Superscript: model
• Subscript: the layer of the model.

Experiments/Results

I initially assumed image workloads, as my own interests are in image processing. To your (NicoLab28) credit, you already showed text workloads should be affine. So for test data I used the COCO 2017 image set. In 3 phases: first 120 random images, ensuring I used each category, then 240, then finally all 5000 images in the COCO data set. The whole run was around 4 hours on an H200.

Based on the first 120

Results:

• 8b L1/2/3 <=> 32B L1/2/3 (~0.6): Ignored, as this gives no real speedup.
• 8b L16/17 <=> 32B L29 (~0.6)
• 8b L16/17 <=> 32B L35/36/37 (~0.4)

L16 was always ahead of L17.

The L35/L36/L37 "plateau" peaked at L36.

There were a small number of outliers (<5%) that showed CKA of ~0, so they were ignored.

Based on the next 240

I repeated the full layer-pair sweep on the next 240 images. The same overall result held, with caveats. If you ignored values with a CKA ~0 (@ 8bL16/32bL29), then the results were confirmed.

CKA ~0 (@ 8bL16/32bL29) outliers now became a substantial minority, representing more than 25% of the sample set. This could not be ignored.

"Good Images" (CKA >= 0.4 @ 8bL16/32bL29) follow a fairly predictable path: their CKA drops off (8bL16/32bL30, 8bL16/32bL31, ..), recovers for a small spike around L35/L36/L37, then reaches a minimum at L44. "Bad Images" (CKA < 0.1 @ 8bL16/32bL29) don't follow this pattern. They effectively flatline at 0 and remain very near 0. They refuse to show any shared geometry.

Fuck it, run all 5000

I ran the full layer-pair sweep for the entire 5000-image COCO 2017 data set (including my previous test images). Previous statements about layer-by-layer correlations hold for "Good Images".

"Bad Images" are no longer a minority; now they're about 50% of the data set. This is the Bimodality I previously discussed. It was almost a perfect 50/50 split, with around 2600/5000 images being "good" and the rest being "bad". All statements from the previous 240 test run still hold about layer-to-layer behavior across a much larger sample set.

Digging deeper, "Bad Images" had fundamentally different hidden-state vector activation-energy concentrations across all "tokens" for a given image input. 8b had a tendency, for "some inputs", to concentrate the activation energy in a single hidden-state vector. In layman's terms, if an image was, for example, 150 tokens of input, effectively only "one" token (or its representational hidden-state vector) would dominate by 8bL16 for "bad images". 32b never did this; the per-token hidden-state norms generally stayed within a few percent of one another. This mimicked how "good images" in 8b behaved.

I found myself thinking:

8b is a smaller model. So possibly for some images/objects/subjects/etc. it can collapse their semantic vector space into a higher-density representation. Effectively 'ignoring' some hidden-state vectors (e.g.: token inputs) as they are superfluous to the internal representation, and the larger model (32b) wasn't forced into the "lower-entropy state" due to its higher parameter count.

My hypothesis crystallized as:

Does semantic state collapse for certain subjects/objects?

Nope. I could find no correlation between resolution, COCO category, blurriness/sharpness, token length, color channel, luminance, subject, number of subjects; absolutely nothing correlated. I want to be clear here, because our human intuition/pattern matching is going to cause you to jump to conclusions which I have disproved: Imagine 4 pictures of a surfboard; 2 will have "good" shared geometry and 2 will be overly concentrated "bad" geometry. Why? IDFK

What about text?

Basically affine. A small ridge regression confirmed this. After less than an hour of training, pure-text workloads were functional.

So why not an affine conversion

This document gets into it

The converter has to "understand" that 8b will (erroneously compared to 32b) concentrate activation energy on some hidden-state vectors, and it has to recognize this is occurring and "spread" that activation energy out to nearby neighboring vectors. You can almost think of this as an "inverse attention pass" (in non-technical terms), as 8b concentrated too much attention (activation energy) in 1 hidden-state vector. This has to be corrected before being handed off to anything pretending to be qwen3-vl-32b or an H3 preconditioning.

On the bimodal "bad images" — that sounds like attention sinks rather than a property of the images.

I hit the same thing measuring the effective rank of the H3 conditioning target. Token 0 came out at norm 15,944 against a median of 294 across the sequence. Until I filtered it, the measurement claimed a single direction carried 90% of the variance, which is obviously wrong. Dropping tokens whose norm exceeds ~5x the sequence median put it back at 3,255 directions for 99% of the energy.

A CKA computed over activations where one token sits 50x above the rest is not measuring representational alignment any more, it is measuring whether the two models put their sink in the same place. Which would explain both things you noticed: the bimodality, and the fact that no image property predicts it — sink position depends on the sequence, not on what the image contains.

Worth re-running your layer sweep with a norm filter before the CKA. If the bad half collapses, the layer pair you settled on may not be the best one — you would have been optimising partly on sink agreement. It might also move the ~0.6 you measured quite a bit.

One caution from my side: high CKA did not imply a linear map exists. I measured 0.95 CKA between a 4B and the 32B target and still could not get a plain linear projection past 0.70 cosine. CKA says the two models organise information the same way; it does not say a matrix can carry you from one to the other.

One caution from my side: high CKA did not imply a linear map exists

wtf? The model published here isn't linear. The previous comment has a literally header line/subject dedicated to why an affine/linear conversion doesn't work and a link to a write up detailing the experiment that showed this doesn't.

You're warning me to not do something, I did not do?

This comment shows an extreme lack of comprehension and context to a degree that the most charitable assumption I can make is you're blindly copying & pasting these comments into an AI chat.

You're right, I was talking nonsense.
I was probably still too focused on my own pipeline when I wrote that, and I was trying to spare you a problem you wouldn't even encounter.
Sorry for the unnecessary noise.