dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta

Fock-PARFLM v2.1 with Depth-Conditioned Anisotropic Multi-Context Gaussian V_theta (TinyStories)

The anisotropic sibling of the depth-conditioned multi-context Gaussian V_theta checkpoint. Both models share the identical Fock-PARFLM v2.1 scaffold (d=256, L=8, 4 xi-context channels, 16 Fock registers) and the identical bounded-mixture-of-Gaussian-wells design for $V_{\theta }V\_\backslash theta$, differing in exactly one structural choice: this model's precision matrix per well is diagonal + a learned rank-4 low-rank correction (Sigma_k^{-1} = diag(a_k) + B_k B_k^T) instead of purely diagonal, giving each well a non-axis-aligned, ellipsoidal shape rather than an axis-aligned one. Training also adds a small auxiliary Fock-coupling log-barrier regulariser on the xi-channel coupling strengths, which the isotropic sibling did not use.

The effect of this one change is large: honest validation PPL drops from 16.33 (isotropic) to 9.04 (anisotropic) — a 7.29 PPL improvement, and, notably, the first time any bounded-potential Fock-PARFLM variant on TinyStories has overtaken the unbounded MLP (9.70) and SQ3 (10.90) baselines rather than trailing them. See When to Use This Model and Why the Anisotropic Correction Wins for the full comparison and the reasons behind the gap.

Like its isotropic sibling, this checkpoint needed no causal-leak re-training: it was trained from step 0 with prefix_causal_registers=True, and its bit-exact future-perturbation probes and honest-vs-standard PPL checks pass cleanly throughout training. See Training Details for the probe evidence.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Model
• Architecture
• The Analytical Form of the Potential
• The Fock-Coupling Regularisation Term
• Why the Anisotropic Correction Wins
• How to Get Started
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

When to Use This Model

Choose this variant over the isotropic Gaussian, SQ3-structured, or MLP Fock-PARFLM v2.1 variants when you want the best TinyStories PPL currently published for this family while keeping a structurally bounded, interpretable potential:

Priority	Aniso-Gaussian + fock-reg (this model)	Isotropic Gaussian (sibling)	SQ3 structured (sibling)	MLP (baseline)
Boundedness	V_theta in [-4, 0] by construction, force magnitude capped	Same bound	Unbounded (log-sum-exp mixture)	Unbounded
Well shape	Ellipsoidal, non-axis-aligned (rank-4 low-rank precision correction)	Axis-aligned only (diagonal precision)	N/A (quadratic mixture)	N/A (black-box)
Interpretability	32 explicit attractor centres, one bank per temporal horizon	Same	8 explicit attractor centres	Black-box
Per-layer specialisation	Yes — shared bank + learned per-layer depth-code shift	Same	No	No
Raw PPL	9.04	16.33	10.90	9.70
Total parameters	26.6M (larger V_theta hypernetwork)	18.2M	18.2M	17.4M
Causal-leak history	Trained leak-free natively (no re-training needed)	Trained leak-free natively	Re-trained with the fix (+0.54 PPL tax)	Re-trained with the fix (+0.40 PPL tax)

Bottom line: at this same small scale (d=256, L=8, 20k steps) where the isotropic Gaussian trailed the unbounded siblings by a 6.63 PPL gap, the anisotropic correction plus the Fock-coupling regulariser not only close that gap but overtake both unbounded baselines, reaching a new family-best honest TinyStories PPL of 9.04 — 0.66 better than the MLP and 1.86 better than SQ3. This is not a free lunch: the anisotropic $V_{\theta }V\_\backslash theta$ hypernetwork is ~3x larger than the isotropic one (12.6M vs 4.2M parameters), so the comparison is not parameter-matched. See Why the Anisotropic Correction Wins for a fuller discussion of what is and is not driving the improvement.

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps (layer index g = 0..7):
       |
       +-- K-EMA channels: xi^(m)_t = causal_ema(h, alpha_m)   [n_ctx=4 channels]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [4 contexts, 32 wells total]
       |     f_theta = -analytical_grad_h V_theta            [closed-form, bounded]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=16 virtual registers with Q/K/V creation gates
       |     LIFO stack discipline, salience decay
       |     Per-register tau and key subspaces
       |     Reverse channel (non-conservative exchange), prefix-causal (leak-free)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

Parameter	Value
Hidden dim (d)	256
Layers (L)	8
V_theta kind	Depth-conditioned multi-context anisotropic Gaussian (bounded mixture)
V_theta contexts (n_ctx)	4 (one bank per xi channel)
Wells per context (K)	8
Total attractors	32
Anisotropic rank (r)	4 — low-rank factor B_k in R^(256x4) per well
Precision init / cap	a_k init at -log(256) (log-precision), capped at 2/256
Depth codes	per-layer additive shift, shape (L=8, n_ctx=4, d=256), init std 0.02
Xi channels (K_xi)	4
Fock-coupling regulariser	log-barrier on alpha_k, lambda=0.005, eps=1e-6 (training-only, see The Fock-Coupling Regularisation Term)
V_phi kind	structural_competitive
V_phi hidden	128
Top-k (sparse routing)	8
Gumbel tau	1.0 (init), 0.3 (min)
Fock version	v2.1
Registers (M)	16
Register d_k	64
Stack discipline	LIFO
Reverse channel	Yes — prefix-causal, leak-free
Per-register tau/keys	Yes
Gathered V_phi	Yes
Per-layer V_phi scale	Yes
LN before distance	Yes
Layer checkpoint	Yes
Mass model	logfreq (frozen surprisal lookup)
Damping gamma	0.30 (see Evaluation Results for a gamma=0.15 ablation)
lambda_V (V_theta regularisation)	0.01
Prefix-causal registers	Yes — trained natively with the fix, not retrofitted
Total parameters	26,635,275
V_theta parameters	12,648,480

The Analytical Form of the Potential

Each of the 4 xi-context channels gets its own bank of 8 wells, exactly as in the isotropic sibling. The only change is the precision structure: instead of a purely diagonal precision $a_k\in {\mathbb{R}}^{256}a\_k\; \backslash in\; \backslash mathbb\{R\}^\{256\}$, each well now carries an additional learned low-rank factor $B_k\in {\mathbb{R}}^{256\times 4}B\_k\; \backslash in\; \backslash mathbb\{R\}^\{256\; \backslash times\; 4\}$, giving an effective inverse-covariance

$${\mathrm{\Sigma }}_k^{-1}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(a_k)+B_k{B}_k^{\mathrm{\top }},B_k\in {\mathbb{R}}^{256\times 4},a_k>0.\backslash Sigma\_k^\{-1\}\; =\; \backslash mathrm\{diag\}(a\_k)\; +\; B\_k\; B\_k^\backslash top,\; \backslash qquad\; B\_k\; \backslash in\; \backslash mathbb\{R\}^\{256\; \backslash times\; 4\},\; \backslash quad\; a\_k\; >\; 0.$$

With $\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}=h-{\mu }_k^{(m)}({\xi }^{(m)})\backslash mathrm\{diff\}\backslash \_k^\{(m)\}\; =\; h\; -\; \backslash mu\_k^\{(m)\}(\backslash xi^\{(m)\})$, context $mm$'s contribution is

$$V_m({\xi }^{(m)},h)=-\sum _{k=1}^8{w}_k^{(m)}({\xi }^{(m)})\exp (-\frac{1}{2}[a_k^{(m)}\cdot (\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{)}^2+\parallel B_k^{(m)\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{\parallel }^2])V\_m(\backslash xi^\{(m)\},\; h)\; =\; -\backslash sum\_\{k=1\}^\{8\}\; w\_k^\{(m)\}(\backslash xi^\{(m)\})\; \backslash exp\backslash Big(-\backslash frac\{1\}\{2\}\backslash big[a\_k^\{(m)\}\; \backslash cdot\; (\backslash mathrm\{diff\}\backslash \_k^\{(m)\})^2\; +\; \backslash lVert\; B\_k^\{(m)\backslash top\}\; \backslash mathrm\{diff\}\backslash \_k^\{(m)\}\; \backslash rVert^2\backslash big]\backslash Big)$$

where the first bracket term is the same axis-aligned quadratic form as the isotropic sibling, and the second, new term $\parallel B_k^{\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}{\parallel }^2\backslash lVert\; B\_k^\backslash top\; \backslash mathrm\{diff\}\; \backslash rVert^2$ adds curvature along the (up to) 4 directions spanned by $B_{k}B\_k$'s columns — turning each well from an axis-aligned hyper-ellipsoid into a general ellipsoid whose principal axes are learned rather than fixed to the coordinate frame. Setting rank $r=0r=0$ recovers the isotropic potential exactly (the low-rank term vanishes identically), so the two siblings are nested designs, not unrelated architectures.

Depth conditioning works exactly as in the isotropic sibling: a single shared bank per context is reused across all 8 layers, and each layer $gg$ learns a small additive depth code $e_g^{(m)}\in {\mathbb{R}}^{256}e\_g^\{(m)\}\; \backslash in\; \backslash mathbb\{R\}^\{256\}$ that shifts the context before the bank sees it, ${\xi }_g^{(m)}={\xi }^{(m)}+e_g^{(m)}\backslash xi\_g^\{(m)\}\; =\; \backslash xi^\{(m)\}\; +\; e\_g^\{(m)\}$, with $V_{\theta }(\xi ,h)={\sum }_{m=1}^4{V}_m({\xi }_g^{(m)},h)V\_\backslash theta(\backslash xi,\; h)\; =\; \backslash sum\_\{m=1\}^\{4\}\; V\_m(\backslash xi\_g^\{(m)\},\; h)$.

Force (closed form):

$$f_{\theta }=-{\mathrm{\nabla }}_h{V}_{\theta }=\sum _{m=1}^4\sum _{k=1}^8{w}_k^{(m)}[a_k^{(m)}\odot \mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}+B_k^{(m)}(B_k^{(m)\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)})]\exp (-\frac{1}{2}[a_k^{(m)}\cdot (\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{)}^2+\parallel B_k^{(m)\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{\parallel }^2])f\_\backslash theta\; =\; -\backslash nabla\_h\; V\_\backslash theta\; =\; \backslash sum\_\{m=1\}^\{4\}\backslash sum\_\{k=1\}^\{8\}\; w\_k^\{(m)\}\backslash big[a\_k^\{(m)\}\; \backslash odot\; \backslash mathrm\{diff\}\backslash \_k^\{(m)\}\; +\; B\_k^\{(m)\}(B\_k^\{(m)\backslash top\}\backslash mathrm\{diff\}\backslash \_k^\{(m)\})\backslash big]\backslash exp\backslash Big(-\backslash frac\{1\}\{2\}\backslash big[a\_k^\{(m)\}\backslash cdot(\backslash mathrm\{diff\}\backslash \_k^\{(m)\})^2\; +\; \backslash lVert\; B\_k^\{(m)\backslash top\}\backslash mathrm\{diff\}\backslash \_k^\{(m)\}\backslash rVert^2\backslash big]\backslash Big)$$

The first term inside the brackets, $a_k\odot \mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}ka\_k\; \backslash odot\; \backslash mathrm\{diff\}\backslash \_k$, is the same axis-aligned pull as the isotropic sibling; the second term, $B_k(B_k^{\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}k)B\_k(B\_k^\backslash top\; \backslash mathrm\{diff\}\backslash \_k)$, adds a pull component along the directions $B_{k}B\_k$ defines — this is what lets a single well exert direction-selective force (e.g. a strong pull along one axis combined with a weak or even negligible pull along a nearby one), something an axis-aligned diagonal precision cannot represent. The same Gaussian decay factor that bounds the isotropic force also bounds this one: since $a_k>0a\_k\; >\; 0$ (softplus) and $B_k{B}_k^{\mathrm{\top }}B\_k\; B\_k^\backslash top$ is positive semi-definite, the exponent argument is never positive, so every well's magnitude still peaks at a finite radius and vanishes both at the centre and far away.

Boundedness is unaffected by anisotropy. Because the exponential factor is still bounded in $(0,1](0,\; 1]$, each well's contribution is still bounded in $[-1,0][-1,\; 0]$, each context's contribution in $[-1,0][-1,\; 0]$, and the full $V_{\theta }V\_\backslash theta$ (summed over 4 contexts) in $[-4,0][-4,\; 0]$ — identical to the isotropic sibling's theoretical bound. The low-rank correction changes the shape of the wells, not the range of the potential.

Empirical landscape. On this checkpoint the observed $V_{\theta }V\_\backslash theta$ range over validation batches is [-0.0102, -0.0000024] (mean -0.0000024, std 0.0001) — even smaller in raw magnitude than the isotropic sibling's [-0.624, -0.00002] range. This is a useful and slightly counter-intuitive data point: the PPL improvement over the isotropic sibling is not coming from a larger swing in the potential's value. See Why the Anisotropic Correction Wins for what is actually driving it.

For the full derivation, boundedness proof, and the base isotropic design, see the companion note: Structured_VTheta_Design_and_Theory.md and the isotropic sibling's analytical form section.

The Fock-Coupling Regularisation Term

This checkpoint's training loss carries one auxiliary term the isotropic sibling's does not: a log-barrier penalty on the xi-channels' coupling strengths — how strongly each of the 4 causal context channels stays coupled to its own past, as opposed to collapsing into a copy of the instantaneous hidden state.

Where ${\alpha }_k\backslash alpha\_k$ comes from. Each xi-context channel is a causal, weighted exponential moving average (EMA) of the hidden state (see the MLP sibling for the base multi-channel-xi design):

$${\xi }_k[t]=\sum _{s\le t}{W}_k[t,s]h[s],W_k[t,s]=\frac{{\alpha }_k^{t-s}}{Z_k[t]},Z_k[t]=\sum _{r\le t}{\alpha }_k^{t-r},\backslash xi\_k[t]\; =\; \backslash sum\_\{s\; \backslash le\; t\}\; W\_k[t,s]\; h[s],\; \backslash qquad\; W\_k[t,s]\; =\; \backslash frac\{\backslash alpha\_k^\{t-s\}\}\{Z\_k[t]\},\; \backslash qquad\; Z\_k[t]\; =\; \backslash sum\_\{r\; \backslash le\; t\}\; \backslash alpha\_k^\{t-r\},$$

where the decay ${\alpha }_k=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}(z_k)\in (0,1)\backslash alpha\_k\; =\; \backslash mathrm\{sigmoid\}(z\_k)\; \backslash in\; (0,1)$ is a learned, unconstrained parameter $z_{k}z\_k$ passed through a sigmoid. Two limits matter here: as ${\alpha }_k\to 0\backslash alpha\_k\; \backslash to\; 0$, the weight matrix concentrates entirely on $s=ts=t$, so ${\xi }_k[t]\to h[t]\backslash xi\_k[t]\; \backslash to\; h[t]$ — the channel degenerates into an instantaneous copy of the hidden state, contributing nothing that $V_{\theta }V\_\backslash theta$ doesn't already see directly through $hh$ itself. As ${\alpha }_k\to 1\backslash alpha\_k\; \backslash to\; 1$, the weights approach an unweighted running mean over the whole causal prefix — maximal memory.

The regulariser. The auxiliary loss is a log-barrier on each channel's coupling strength:

$$L_{\text{fock}}(\theta )=-{\lambda }_{\text{fock}}\sum _{k=1}^4\log ({\alpha }_k+ϵ),{\lambda }_{\text{fock}}=0.005,ϵ={10}^{-6},L\_\{\backslash text\{fock\}\}(\backslash theta)\; =\; -\backslash lambda\_\{\backslash text\{fock\}\}\; \backslash sum\_\{k=1\}^\{4\}\; \backslash log(\backslash alpha\_k\; +\; \backslash epsilon),\; \backslash qquad\; \backslash lambda\_\{\backslash text\{fock\}\}\; =\; 0.005,\; \backslash quad\; \backslash epsilon\; =\; 10^\{-6\},$$

added directly to the existing loss (next-token cross-entropy plus the $V_{\theta }V\_\backslash theta$ magnitude penalty):

$$L_{\text{total}}=L_{\text{ntp}}+{\lambda }_V{R}_V+L_{\text{fock}}(\theta )\mathrm{.}L\_\{\backslash text\{total\}\}\; =\; L\_\{\backslash text\{ntp\}\}\; +\; \backslash lambda\_V\; R\_V\; +\; L\_\{\backslash text\{fock\}\}(\backslash theta).$$

Because $-\log ({\alpha }_k+ϵ)\to +\mathrm{\infty }-\backslash log(\backslash alpha\_k+\backslash epsilon)\; \backslash to\; +\backslash infty$ as ${\alpha }_k\to 0\backslash alpha\_k\; \backslash to\; 0$ and $\to 0\backslash to\; 0$ as ${\alpha }_k\to 1\backslash alpha\_k\; \backslash to\; 1$, this is a one-sided barrier: it penalises channels drifting toward the degenerate instantaneous limit and is essentially inert for channels already well-coupled to their causal history.

How it interacts with the rest of the model. The optimiser does not update ${\alpha }_k\backslash alpha\_k$ directly — it updates the unconstrained $z_{k}z\_k$, with ${\alpha }_k=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}(z_k)\backslash alpha\_k\; =\; \backslash mathrm\{sigmoid\}(z\_k)$. Differentiating through the sigmoid,

$$\frac{\mathrm{\partial }{L}_{\text{fock}}}{\mathrm{\partial }{z}_k}=-{\lambda }_{\text{fock}}\frac{{\alpha }_k(1-{\alpha }_k)}{{\alpha }_k+ϵ}\mathrm{.}\backslash frac\{\backslash partial\; L\_\{\backslash text\{fock\}\}\}\{\backslash partial\; z\_k\}\; =\; -\backslash lambda\_\{\backslash text\{fock\}\}\; \backslash frac\{\backslash alpha\_k(1-\backslash alpha\_k)\}\{\backslash alpha\_k\; +\; \backslash epsilon\}.$$

For ${\alpha }_k\gg ϵ\backslash alpha\_k\; \backslash gg\; \backslash epsilon$ this reduces to $\approx -{\lambda }_{\text{fock}}(1-{\alpha }_k)\backslash approx\; -\backslash lambda\_\{\backslash text\{fock\}\}(1-\backslash alpha\_k)$, which has two notable properties:

• It does not vanish as ${\alpha }_k\to 0\backslash alpha\_k\; \backslash to\; 0$. It approaches a constant, $-{\lambda }_{\text{fock}}-\backslash lambda\_\{\backslash text\{fock\}\}$ — a persistent push toward larger $z_{k}z\_k$ that does not weaken even for very small ${\alpha }_k\backslash alpha\_k$. This is worth highlighting because a naive log-barrier applied directly to a sigmoid output usually does suffer vanishing gradients near the boundary (the sigmoid's own derivative ${\alpha }_k(1-{\alpha }_k)\backslash alpha\_k(1-\backslash alpha\_k)$ shrinks to 0 there); here the barrier's own blow-up as ${\alpha }_k\to 0\backslash alpha\_k\; \backslash to\; 0$ exactly cancels that shrinkage, so the correction stays effective for however small ${\alpha }_k\backslash alpha\_k$ drifts (until it gets close enough to $ϵ={10}^{-6}\backslash epsilon\; =\; 10^\{-6\}$ that the $+ϵ+\backslash epsilon$ floor takes over).
• It vanishes smoothly as ${\alpha }_k\to 1\backslash alpha\_k\; \backslash to\; 1$. Once a channel is well-coupled to its history, the term contributes essentially zero gradient, leaving the language-modelling loss free to place ${\alpha }_k\backslash alpha\_k$ wherever it wants above the floor.

In effect, $L_{\text{fock}}L\_\{\backslash text\{fock\}\}$ acts as a floor that keeps all 4 xi-channels feeding genuinely distinct temporal contexts into the four anisotropic Gaussian well banks, rather than letting the optimiser quietly collapse one or more channels into a redundant copy of $hh$ — which would waste an entire bank's worth of $V_{\theta }V\_\backslash theta$ capacity (8 wells, ~2.1M parameters) on an input carrying no information the rest of the model doesn't already have. This term only touches the xi-channel decays $z_{k}z\_k$; it does not directly touch $V_{\theta }V\_\backslash theta$'s well parameters (\(\mu_k, a_k, B_k, w_k\)) or any other part of the model — its effect on those is entirely indirect, mediated through the richer, more temporally-diverse $\xi \backslash xi$ inputs the wells receive once channel collapse is prevented. The isotropic sibling has no such floor; empirically, this checkpoint's final decays $[{\alpha }_1,\dots ,{\alpha }_4]=[0.418,0.609,0.788,0.923][\backslash alpha\_1,\backslash ldots,\backslash alpha\_4]\; =\; [0.418,\; 0.609,\; 0.788,\; 0.923]$ are comfortably spread across the memory-horizon range with none collapsed toward 0, consistent with the barrier doing its intended job (see Learned Xi-Channel Decay Rates).

Why the Anisotropic Correction Wins

Three changes distinguish this checkpoint from the isotropic sibling, and all three plausibly contribute to the 7.29 PPL gap:

1. Direction-selective force, not just axis-aligned force. The low-rank term $B_k(B_k^{\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}k)B\_k(B\_k^\backslash top\; \backslash mathrm\{diff\}\backslash \_k)$ lets a single well pull $hh$ toward its centre along directions that are not aligned with the coordinate axes of the residual stream. An axis-aligned diagonal precision can only stretch or compress each of the 256 dimensions independently; the rank-4 correction adds up to 4 additional, jointly-learned directions per well along which curvature can concentrate. Given that the empirical $V_{\theta }V\_\backslash theta$ magnitude is smaller than the isotropic sibling's, this suggests the benefit comes primarily from the direction of the force (which shapes the trajectory of $hh$ through the integration steps) rather than its magnitude — a small but well-aimed push can matter more than a larger but axis-locked one.

2. The Fock-coupling regulariser keeps all 4 xi-channels engaged. The isotropic sibling's wells were only weakly activated (std 0.024 on a [-4,0] scale) with no mechanism to prevent the xi-channel coupling strengths ${\alpha }_k\backslash alpha\_k$ from drifting toward the degenerate, instantaneous corner. See The Fock-Coupling Regularisation Term for the exact loss term and why its gradient stays effective even for small ${\alpha }_k\backslash alpha\_k$ rather than vanishing. The final learned values, $[{\alpha }_1,\dots ,{\alpha }_4]=[0.418,0.609,0.788,0.923][\backslash alpha\_1,\backslash ldots,\backslash alpha\_4]\; =\; [0.418,\; 0.609,\; 0.788,\; 0.923]$, are comfortably spread and none has collapsed — consistent with all 4 context banks staying informative throughout training.

3. A substantially larger $V_{\theta }V\_\backslash theta$ hypernetwork. The low-rank factor $B_{k}B\_k$ is itself produced by a learned projection (B_proj, mapping the 256-dim context into K x d x rank = 8192 numbers per bank), which adds roughly 8.4M parameters across the 4 banks. Total $V_{\theta }V\_\backslash theta$ parameters rise from 4.2M (isotropic) to 12.6M (anisotropic), and total model parameters from 18.2M to 26.6M. This comparison is therefore not parameter-matched against the isotropic sibling, the SQ3 sibling, or the MLP baseline — part of the gap may simply reflect the larger capacity available to shape the potential, independent of the anisotropy itself. Disentangling "extra capacity" from "anisotropic shape" (e.g. by comparing against an isotropic model with a similarly enlarged hypernetwork) is a natural follow-up ablation not yet run.

A damping ablation run alongside this checkpoint (gamma=0.15, otherwise identical) reached only 10.05 PPL — worse than gamma=0.30's 9.04 — suggesting the larger, more expressive anisotropic $V_{\theta }V\_\backslash theta$ benefits from the stronger damping's stabilising effect on training, consistent with the broader gamma-sweep findings reported elsewhere in the Semantic Simulation companion notes.

How to Get Started

import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing

# -- Build base model --
config = FockMultiXiPARFConfig(
    vocab_size=50257, d=256, max_len=1024,
    L=8, v_hidden=1024, v_depth=3, dt=1.0,
    mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    logfreq_init_alpha=0.1,
    init_gamma=1.0,
    fixed_gamma=0.30,
    causal_force=True,
    ln_after_step=True,
    xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True,
    xi_alpha_init_mode="explicit",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
    fock_version="v2",
    n_registers=16,
    register_salience_decay=0.5,
    register_salience_threshold=0.005,
    creation_gate_hidden=64,
    stack_discipline=True,
    d_k=64,
    tau_create_init=8.0,
    reverse_channel=True,
    per_register_tau=True,
    per_register_keys=True,
    ortho_register_init=True,
    prefix_causal_registers=True,
)
model = FockMultiXiPARFLM(config)

# -- Swap in the depth-conditioned anisotropic multi-context Gaussian V_theta --
import math
model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=256, K=8, n_ctx=4, n_layers=8, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(256),
    precision_max=2.0 / 256,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)  # wires the per-layer index into V_theta's depth-code lookup

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly (32 wells: 4 contexts x 8 wells) --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                                # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)              # (1, 64, 4, 8, 256)
    print(f"Attractor centres shape: {centres.shape}")

Available Artifacts

File	Description
checkpoint/ckpt_best.pt	Best checkpoint (9.04 PPL at step 19,200, gamma=0.30), trained natively leak-free
training_log.jsonl	Per-step training metrics, including causal_probe and trained_leak_probe events
training_curve_aniso_gaussian.png	Training/validation loss curves
v_theta_hist_aniso_gaussian.png	V_theta output distribution histogram
landscape_stats_aniso_gaussian.json	V_theta landscape statistics (mean, std, min, max, range)
model_aniso_gaussian_vtheta.py	Anisotropic Gaussian V_theta classes (single-bank, multi-context, depth-conditioned) + install_aniso_depth_routing
config.json	Model configuration

Training Details

Training Data

TinyStories --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens.

Training Procedure

The base architecture is identical to Fock-PARFLM v2.1 and to the isotropic Gaussian sibling. The only structural modification is the $V_{\theta }V\_\backslash theta$ precision: a single shared AnisotropicDepthConditionedGaussianVTheta bank (4 contexts x 8 wells, rank-4 low-rank correction per well) plus per-layer depth codes are constructed in place of the isotropic bank, before training begins from scratch. The pairwise $V_{\varphi }V\_\backslash phi$ (competitive structural MLP, hidden=128, top-k=8) and Fock register pool (16 registers, LIFO, reverse channel) are unchanged, and prefix_causal_registers=True is set from the start. Training additionally applies the Fock-coupling log-barrier regulariser described in The Fock-Coupling Regularisation Term (lambda=0.005, eps=1e-6), which the isotropic sibling did not use.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	5e-4 (cosine decay)
Warmup steps	400
Gradient clipping	1.0 (0.5 for Fock-register gradients, per-group)
Batch size	16 (4 x grad-accum 4)
Block size	512
Training steps	20,000
lambda_V (V_theta regularisation)	0.01
lambda_fock (coupling regulariser)	0.005
Hardware	NVIDIA A100 40GB (Google Colab)

Causal-Leak Verification (no fix needed — trained natively leak-free)

Because this run started with prefix_causal_registers=True from step 1 (rather than being re-trained after the fact like the MLP and SQ3 siblings), the training log carries the same probes as a matter of routine certification, not remediation:

step	causal_probe_max_delta	causal_probe_passed	honest vs. standard PPL diff (nats)
4,000	0.0	True	—
8,000	0.0	True	+0.0203 ± 0.0196 SE
12,000	0.0	True	—
16,000	0.0	True	+0.0475 ± 0.0244 SE
20,000	0.0	True	—

The deterministic architectural probe is bit-exact 0.0 at every checkpoint, and the honest-vs-standard PPL differences are both within ~2 standard errors of zero — not statistically distinguishable from no leak. No causal-leak caveat applies to the 9.04 PPL reported on this card.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_aniso_gaussian_fockreg_tinystories.ipynb (companion repo) --- Colab notebook with the depth-conditioned anisotropic Gaussian arm, GDrive output, checkpointing, and live progress display. A gamma=0.15 ablation was run from the same notebook; see Why the Anisotropic Correction Wins.

Evaluation Results

TinyStories Validation Perplexity

Model	PPL	Params	Analytical V_theta grad	Bounded V_theta	Causal leak
Matched Attention (baseline)	7.81	19.5M	---	---	No
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, this model)	9.04	26.6M	Yes	Yes	Never leaked
Fock-PARFLM v2.1 (MLP)	9.70	17.4M	No	No	Fixed & verified
Fock-PARFLM v2.1 (SQ3)	10.90	18.2M	Yes	No	Fixed & verified
Fock-PARFLM v2.1 (isotropic Gaussian)	16.33	18.2M	Yes	Yes	Never leaked
Aniso-Gaussian + fock-reg, gamma=0.15 ablation	10.05	26.6M	Yes	Yes	Never leaked

All PPL values in this table are honest, leak-free numbers. This model is the first bounded-\(V_\theta\) Fock-PARFLM variant on TinyStories to beat the unbounded Matched-Attention baseline's near-neighbours (MLP, SQ3) rather than trail them — at the cost of a larger $V_{\theta }V\_\backslash theta$ hypernetwork (26.6M vs 17.4-18.2M total parameters); see Why the Anisotropic Correction Wins.

PPL is evaluated on the TinyStories validation split; see Training Data.

V_theta Landscape Statistics

Metric	This model (aniso-Gaussian)	Isotropic Gaussian	Fock-PARFLM (SQ3)	Fock-PARFLM (MLP)
Mean V_theta	-0.0000024	-0.017	0.017	n/a (MLP has no comparable scalar readout)
Std V_theta	0.0001	0.024	0.42	n/a
Range	0.0102	0.624	16.0	n/a
Theoretical bound	[-4, 0]	[-4, 0]	(-∞, +∞)	(-∞, +∞)

Despite having the best PPL of any Fock-PARFLM variant on TinyStories, this model has the flattest $V_{\theta }V\_\backslash theta$ landscape ever observed in this family — a reminder that raw potential magnitude is not a reliable proxy for how much useful work $V_{\theta }V\_\backslash theta$ is doing; see Why the Anisotropic Correction Wins.

Learned Xi-Channel Decay Rates

The final learned alpha values $[{\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4]=[0.418,0.609,0.788,0.923][\backslash alpha\_1,\; \backslash alpha\_2,\; \backslash alpha\_3,\; \backslash alpha\_4]\; =\; [0.418,\; 0.609,\; 0.788,\; 0.923]$ (initialised at $[0.25,0.5,0.75,0.95][0.25,\; 0.5,\; 0.75,\; 0.95]$) are the most evenly spread of any Fock-PARFLM sibling published so far — consistent with the Fock-coupling regulariser doing its job of keeping every context channel engaged rather than letting the optimiser collapse one or more of them.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Model	Design	PPL	HuggingFace
Multi-Xi SPLM (MLP)	Pure scalar potential	11.51	semsimula-splm-multixi
Multi-Xi PARFLM (MLP)	Scalar + pairwise forces	12.06	semsimula-parflm-multixi
Fock-PARFLM v2.1 (MLP)	PARFLM + Fock registers	9.70	semsimula-fock-parflm
Fock-PARFLM v2.1 (SQ3)	Structured + pairwise + Fock	10.90	semsimula-fock-parflm-structured-vtheta
Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian)	Bounded multi-context + pairwise + Fock	16.33	semsimula-fock-parflm-depthcond-vtheta
Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian)	Bounded, ellipsoidal multi-context + pairwise + Fock	9.04	this model
Fock-PARFLM v2.1 (depth-cond. Gaussian, OpenWebText scale)	Bounded multi-context + pairwise + Fock, d=384 L=16	27.23	semsimula-fock-parflm-depthcond-vtheta-openwebtext
Fock-Attention	Fock + attention	9.42	semsimula-fock-attention
Hybrid SPLM+Attn	Attention + SPLM refinement	8.50	semsimula-hybrid-splm

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Research checkpoint only. This model is a proof-of-concept for bounded, anisotropic, depth-conditioned scalar potentials in Fock-augmented architectures, not a production system.
• TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
• English only. No multilingual capability.
• Not parameter-matched against its siblings. At 26.6M total parameters (12.6M in $V_{\theta }V\_\backslash theta$ alone), this model is ~45% larger than the isotropic Gaussian, SQ3, and MLP siblings (17.4-18.2M each). Part of its PPL advantage may be attributable to the larger $V_{\theta }V\_\backslash theta$ hypernetwork rather than to anisotropy per se; a parameter-matched ablation has not yet been run. See Why the Anisotropic Correction Wins.
• No safety training. No RLHF, DPO, or safety filtering has been applied.
• V_phi and Fock forces still use autograd. Only the $V_{\theta }V\_\backslash theta$ gradient is analytical; the $V_{\varphi }V\_\backslash phi$ pairwise force and Fock register forces still require torch.autograd.grad.
• Extremely flat empirical landscape. The empirical $V_{\theta }V\_\backslash theta$ range (0.0102) is far below the theoretical bound (4.0) and even smaller than the already-flat isotropic sibling's — despite this being the best-performing $V_{\theta }V\_\backslash theta$ variant published so far. This counter-intuitive combination (best PPL, flattest landscape) is discussed in Why the Anisotropic Correction Wins; it should not be read as evidence that the wells are unused, only that their magnitude is not what matters.
• Fock-coupling regulariser is an added training-time term. The auxiliary log-barrier loss on ${\alpha }_k\backslash alpha\_k$ is not present in the isotropic sibling's training, which is a second confound alongside parameter count when attributing the PPL gap to "anisotropy" specifically.
• No causal-leak issue. Like the isotropic sibling, this checkpoint was trained natively with prefix_causal_registers=True and needed no re-training or PPL correction; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: NVIDIA A100 40GB (Google Colab)
• Training time: ~4 hours (20,000 steps)
• Carbon footprint: Estimated less than 3 kg CO2