Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Quazim0t0 
posted an update 6 days ago
Post
2386
Created research language model whose channel-mixing block is not an MLP. It is a differentiable Neighbour-Sensing fungal-colony-growth model: each token is expanded into a colony of hyphal tips that grow in a bounded latent region, sense a shared density field, and steer their own growth — the "MLP" is replaced by a few differentiable steps of colony growth, read back out into the hidden state.

Quazim0t0/Mycel-LM-79M

Also the original SpikeWhale project — the one that sparked all the other SpikeWhale related projects. Every spiking primitive here is hand-written in plain PyTorch: the leaky integrate-and-fire (LIF) neuron dynamics, the fast-sigmoid surrogate gradient, and the backprop-through-time training loop. No snntorch, no spikingjelly, no norse, no bindsnet — the network is a genuine from-scratch SNN.

Quazim0t0/SpikeWhale-SNN-216M
dipankarsarkar

The fungal metaphor is the wrong headline. The real claim is a channel mixer built from a growth process instead of a dense matmul.

An MLP mixes every channel to every channel in one shot. A Neighbour-Sensing colony mixes locally and iteratively, so you have traded a dense op for a spatial prior plus depth-in-time.

That only pays off if the routing stays sparse at inference and does not densify to MLP cost once trained.

Did you measure active-channel fan-out per token against a matched dense MLP at 79M? That number is what tells you whether this is a curiosity or a cheaper mixer.

·
Quazim0t0

Appreciate it, but efficiency wasn't the point, and the post already framed it the way you did: a research LM whose channel-mixing block isn't an MLP, just a few differentiable steps of colony growth read back into the hidden state. The only thing I was testing is whether a weird dynamical system can carry the channel-mixing role and still learn language under a fixed pipeline. "Curiosity or cheaper mixer" is a great question, just a different experiment than the one I ran.

That said, I measured your number: active-channel fan-out (Jacobian participation ratio, /768) was ~469 for the growth block vs ~489 for a matched dense MLP, at 3.86M vs 3.54M params and ~1.07x MACs/token. So you're right, it densifies to roughly MLP cost. The growth math is cheap (O(N·F)), but it's bracketed by dense projections, so it's a hair more expensive, not less. Curiosity, plainly. The version that'd make your point true is a sparsified readback (each tip writing to a local slice of channels, or top-k tip-to-channel routing) so fan-out drops well under d_model. That one I'd like to run. Thanks for pushing on it.
dipankarsarkar

That is the most honest answer I have gotten to that question. 469 vs 489 and 1.07x MACs settles today's version: dense-cost curiosity, not a cheaper mixer.

So the sparsified readback is the whole experiment now. The number I would watch is loss at fixed fan-out: pin active channels near d_model/4 and read off the perplexity you pay. Flat means the growth prior is load-bearing. A cliff means the density was.

Are the tips already spatially clustered enough that a top-k readback is nearly free, or does the routing need its own learned gate?

·
Quazim0t0

Measured it: the readout weights all 96 tips basically uniformly and the tips spread across ~15/16 density regions, so there's no clustering to exploit yet, a top-k readback today would be a cliff and it'd need a learned gate. Makes sense in hindsight, negative autotropism is an anti-clustering force by design, so the thing that makes it a colony is what keeps it dense.

I'll build the second variant (flip in a clustering/positive-tropism term so the growth concentrates) and post the fixed-fan-out perplexity numbers. I've got another experiment running right now, so it'll be a bit, but I'll get to it when I can since I want to see this one too.
In this post