dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-fock-g1

Fock-PARFLM v2.1, Depth-Conditioned Anisotropic Gaussian V_theta — First-Order Ablation (Fock-G1, TinyStories)

Fock-G1 is the first-order (pure gradient-flow) ablation of the depth-conditioned anisotropic multi-context Gaussian V_theta with Fock-coupling regularisation checkpoint (the "second-order anchor"). The two models are otherwise identical — same architecture, same data, same seed, same training budget, same hyperparameters — differing in exactly one thing: the per-layer velocity memory delta = h - h_prev that drives the anchor's damped velocity-Verlet integration step is forced to zero at every layer, collapsing the update to a pure first-order gradient step. There is no velocity buffer and no damping coefficient acting at inference time.

Headline result: at matched architecture, data, and budget, this first-order model reaches 8.95 validation PPL — slightly better than the second-order anchor's 9.04, and well within the pre-registered single-seed noise band ${\mathrm{\Delta }}_{\min }=1.00\backslash Delta\_\{\backslash min\}=1.00$ PPL. More strikingly, a dedicated architecture-level comparison of the two checkpoints' learned $V_{\theta }V\_\backslash theta$ potentials (curvature spectrum, well anisotropy, mixture-weight entropy, attractor spacing, depth-code shape) finds the first-order and second-order potentials closer to each other than two second-order checkpoints trained at different damping values are to each other — i.e. the training-order gap is smaller than the natural hyperparameter-to-hyperparameter noise floor. See First-Order vs. Second-Order: Does the Potential Look Different? for the full comparison, including why this result does not generalise automatically to other corpora.

Like the anchor, this checkpoint needed no causal-leak re-training: it was trained from step 0 with prefix_causal_registers=True, and its bit-exact future-perturbation probes and honest-vs-standard PPL checks pass cleanly throughout training. See Training Details.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Model
• Architecture
• The Analytical Form of the Potential
• The Fock-Coupling Regularisation Term
• First-Order vs. Second-Order: Does the Potential Look Different?
• How to Get Started
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

When to Use This Model

Choose this variant over the second-order anchor when a simpler integrator is worth more to you than a small amount of theoretical dynamical richness, and you are working in a similarly small, well-behaved training regime (TinyStories-scale, not yet validated at OpenWebText scale):

Priority	Fock-G1 first-order (this model)	Second-order anchor (sibling)
Dynamics order	First-order gradient flow: h_new = LN(h + beta*f)	Second-order damped velocity-Verlet: h_new = LN(h + delta/(1+dt*gamma) + beta*f)
State carried between layers	h only	h and the implicit velocity memory delta = h - h_prev
Raw PPL	8.95	9.04
V_theta potential shape	Practically indistinguishable from the anchor at this scale/corpus — see dedicated section	—
Total parameters	26.6M (identical architecture, only the integrator differs)	26.6M
Causal-leak history	Trained leak-free natively	Trained leak-free natively
Validated at OpenWebText scale	Not yet — pending	Not yet — pending

Bottom line: on TinyStories, at this scale, first-order training reaches the same operating point as second-order training, both in loss and (to within measurement resolution) in the shape of the learned potential. This is a genuinely interesting, corpus- and scale-dependent result, not a general claim that the inertial term is dispensable — see First-Order vs. Second-Order: Does the Potential Look Different? for the evidence this is not a foregone conclusion at larger scale or on richer corpora.

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps (layer index g = 0..7):
       |
       +-- K-EMA channels: xi^(m)_t = causal_ema(h, alpha_m)   [n_ctx=4 channels]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [4 contexts, 32 wells total]
       |     f_theta = -analytical_grad_h V_theta            [closed-form, bounded]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=16 virtual registers with Q/K/V creation gates
       |     LIFO stack discipline, salience decay
       |     Per-register tau and key subspaces
       |     Reverse channel (non-conservative exchange), prefix-causal (leak-free)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- First-order gradient step (Fock-G1): h += beta*f
       |     beta = dt^2 / (m_b * (1 + dt*gamma_star))     [NO velocity memory: delta := 0]
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

Parameter	Value
Hidden dim (d)	256
Layers (L)	8
V_theta kind	Depth-conditioned multi-context anisotropic Gaussian (bounded mixture)
V_theta contexts (n_ctx)	4 (one bank per xi channel)
Wells per context (K)	8
Total attractors	32
Anisotropic rank (r)	4 — low-rank factor B_k in R^(256x4) per well
Precision init / cap	a_k init at -log(256) (log-precision), capped at 2/256
Depth codes	per-layer additive shift, shape (L=8, n_ctx=4, d=256), init std 0.02
Xi channels (K_xi)	4
Fock-coupling regulariser	log-barrier on alpha_k, lambda=0.005, eps=1e-6 (training-only, see The Fock-Coupling Regularisation Term)
V_phi kind	structural_competitive
V_phi hidden	128
Top-k (sparse routing)	8
Gumbel tau	1.0 (init), 0.3 (min)
Fock version	v2.1
Registers (M)	16
Register d_k	64
Stack discipline	LIFO
Reverse channel	Yes — prefix-causal, leak-free
Per-register tau/keys	Yes
Gathered V_phi	Yes
Per-layer V_phi scale	Yes
LN before distance	Yes
Layer checkpoint	Yes
Mass model	logfreq (frozen surprisal lookup)
Dynamics order	First-order gradient flow — delta := 0 at every layer, no velocity buffer
gamma_star (defines beta only)	0.30 — no active damping at inference; retained purely to parameterise beta = dt^2/(m_b*(1+dt*gamma_star)), see First-Order vs. Second-Order
lambda_V (V_theta regularisation)	0.01
Prefix-causal registers	Yes — trained natively with the fix, not retrofitted
Total parameters	26,635,275 (identical to the second-order anchor)
V_theta parameters	12,648,480

The Analytical Form of the Potential

$V_{\theta }V\_\backslash theta$ is architecturally identical to the second-order anchor's — the first-order ablation changes only the integrator, never the potential's functional form. Each of the 4 xi-context channels has its own bank of 8 wells with a diagonal-plus-low-rank precision:

$${\mathrm{\Sigma }}_k^{-1}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(a_k)+B_k{B}_k^{\mathrm{\top }},B_k\in {\mathbb{R}}^{256\times 4},a_k>0.\backslash Sigma\_k^\{-1\}\; =\; \backslash mathrm\{diag\}(a\_k)\; +\; B\_k\; B\_k^\backslash top,\; \backslash qquad\; B\_k\; \backslash in\; \backslash mathbb\{R\}^\{256\; \backslash times\; 4\},\; \backslash quad\; a\_k\; >\; 0.$$

With $\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}=h-{\mu }_k^{(m)}({\xi }^{(m)})\backslash mathrm\{diff\}\backslash \_k^\{(m)\}\; =\; h\; -\; \backslash mu\_k^\{(m)\}(\backslash xi^\{(m)\})$, context $mm$'s contribution is

$$V_m({\xi }^{(m)},h)=-\sum _{k=1}^8{w}_k^{(m)}({\xi }^{(m)})\exp (-\frac{1}{2}[a_k^{(m)}\cdot (\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{)}^2+\parallel B_k^{(m)\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{\parallel }^2]),V\_m(\backslash xi^\{(m)\},\; h)\; =\; -\backslash sum\_\{k=1\}^\{8\}\; w\_k^\{(m)\}(\backslash xi^\{(m)\})\; \backslash exp\backslash Big(-\backslash frac\{1\}\{2\}\backslash big[a\_k^\{(m)\}\; \backslash cdot\; (\backslash mathrm\{diff\}\backslash \_k^\{(m)\})^2\; +\; \backslash lVert\; B\_k^\{(m)\backslash top\}\; \backslash mathrm\{diff\}\backslash \_k^\{(m)\}\; \backslash rVert^2\backslash big]\backslash Big),$$

and $V_{\theta }(\xi ,h)={\sum }_{m=1}^4{V}_m({\xi }_g^{(m)},h)V\_\backslash theta(\backslash xi,\; h)\; =\; \backslash sum\_\{m=1\}^\{4\}\; V\_m(\backslash xi\_g^\{(m)\},\; h)$ with per-layer depth-conditioning ${\xi }_g^{(m)}={\xi }^{(m)}+e_g^{(m)}\backslash xi\_g^\{(m)\}\; =\; \backslash xi^\{(m)\}\; +\; e\_g^\{(m)\}$, exactly as in the anchor. The closed-form force is

$$f_{\theta }=-{\mathrm{\nabla }}_h{V}_{\theta }=\sum _{m=1}^4\sum _{k=1}^8{w}_k^{(m)}[a_k^{(m)}\odot \mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}+B_k^{(m)}(B_k^{(m)\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)})]\exp (-\frac{1}{2}[a_k^{(m)}\cdot (\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{)}^2+\parallel B_k^{(m)\mathrm{\top }}\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{\_}{k}^{(m)}{\parallel }^2]),f\_\backslash theta\; =\; -\backslash nabla\_h\; V\_\backslash theta\; =\; \backslash sum\_\{m=1\}^\{4\}\backslash sum\_\{k=1\}^\{8\}\; w\_k^\{(m)\}\backslash big[a\_k^\{(m)\}\; \backslash odot\; \backslash mathrm\{diff\}\backslash \_k^\{(m)\}\; +\; B\_k^\{(m)\}(B\_k^\{(m)\backslash top\}\backslash mathrm\{diff\}\backslash \_k^\{(m)\})\backslash big]\backslash exp\backslash Big(-\backslash frac\{1\}\{2\}\backslash big[a\_k^\{(m)\}\backslash cdot(\backslash mathrm\{diff\}\backslash \_k^\{(m)\})^2\; +\; \backslash lVert\; B\_k^\{(m)\backslash top\}\backslash mathrm\{diff\}\backslash \_k^\{(m)\}\backslash rVert^2\backslash big]\backslash Big),$$

bounded in $[-4,0][-4,\; 0]$ exactly as in the anchor (the boundedness proof depends only on the Gaussian mixture's shape, not on which integrator consumes $f_{\theta }f\_\backslash theta$).

Empirical landscape. On this checkpoint the observed $V_{\theta }V\_\backslash theta$ range over validation batches is [-0.01654, -0.0] (mean -0.0000036, std 0.000168) — modestly wider than the second-order anchor's [-0.0102, -0.0000024] range (about 1.6x the range, 1.7x the std), though both are extremely flat relative to the [-4, 0] theoretical bound. See First-Order vs. Second-Order for why this raw-output-scale difference does not carry over to the architecture-level shape comparison.

For the full derivation and the base isotropic design, see the companion note: Structured_VTheta_Design_and_Theory.md and the anchor's analytical form section.

The Fock-Coupling Regularisation Term

Training uses the identical log-barrier regulariser on the xi-channels' EMA decay rates ${\alpha }_k=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}(z_k)\backslash alpha\_k\; =\; \backslash mathrm\{sigmoid\}(z\_k)$ as the anchor — this is a loss-side, architecture-independent term that has nothing to do with the integrator order:

$$L_{\text{fock}}(\theta )=-{\lambda }_{\text{fock}}\sum _{k=1}^4\log ({\alpha }_k+ϵ),{\lambda }_{\text{fock}}=0.005,ϵ={10}^{-6},L\_\{\backslash text\{fock\}\}(\backslash theta)\; =\; -\backslash lambda\_\{\backslash text\{fock\}\}\; \backslash sum\_\{k=1\}^\{4\}\; \backslash log(\backslash alpha\_k\; +\; \backslash epsilon),\; \backslash qquad\; \backslash lambda\_\{\backslash text\{fock\}\}\; =\; 0.005,\; \backslash quad\; \backslash epsilon\; =\; 10^\{-6\},$$

added to the total loss $L_{\text{total}}=L_{\text{ntp}}+{\lambda }_V{R}_V+L_{\text{fock}}(\theta )L\_\{\backslash text\{total\}\}\; =\; L\_\{\backslash text\{ntp\}\}\; +\; \backslash lambda\_V\; R\_V\; +\; L\_\{\backslash text\{fock\}\}(\backslash theta)$. Differentiating through the sigmoid parameterisation gives

$$\frac{\mathrm{\partial }{L}_{\text{fock}}}{\mathrm{\partial }{z}_k}=-{\lambda }_{\text{fock}}\frac{{\alpha }_k(1-{\alpha }_k)}{{\alpha }_k+ϵ},\backslash frac\{\backslash partial\; L\_\{\backslash text\{fock\}\}\}\{\backslash partial\; z\_k\}\; =\; -\backslash lambda\_\{\backslash text\{fock\}\}\; \backslash frac\{\backslash alpha\_k(1-\backslash alpha\_k)\}\{\backslash alpha\_k\; +\; \backslash epsilon\},$$

a gradient that does not vanish as ${\alpha }_k\to 0\backslash alpha\_k\; \backslash to\; 0$ (approaching the constant $-{\lambda }_{\text{fock}}-\backslash lambda\_\{\backslash text\{fock\}\}$) and vanishes smoothly as ${\alpha }_k\to 1\backslash alpha\_k\; \backslash to\; 1$, keeping all 4 xi-channels feeding genuinely distinct temporal contexts into the wells rather than letting the optimiser collapse one into a redundant copy of $hh$. See the anchor's full discussion for the complete derivation.

This checkpoint's final learned decays, $[{\alpha }_1,\dots ,{\alpha }_4]=[0.418,0.611,0.792,0.923][\backslash alpha\_1,\backslash ldots,\backslash alpha\_4]\; =\; [0.418,\; 0.611,\; 0.792,\; 0.923]$, are essentially identical to the anchor's $[0.418,0.609,0.788,0.923][0.418,\; 0.609,\; 0.788,\; 0.923]$ (initialised the same, $[0.25,0.5,0.75,0.95][0.25,\; 0.5,\; 0.75,\; 0.95]$) — a first, simple piece of evidence that the first-order and second-order training trajectories converge to very similar operating points, expanded on below.

First-Order vs. Second-Order: Does the Potential Look Different?

This is the question this checkpoint exists to answer. Two models — architecturally identical apart from the integrator — were trained from the same seed, on the same data, for the same 20,000 steps, and compared along two axes: final loss, and the shape of the learned $V_{\theta }V\_\backslash theta$ potential.

The integrator difference

The second-order anchor's per-layer update carries an implicit velocity memory $\delta =h-h_{\text{prev}}\backslash delta\; =\; h\; -\; h\_\{\backslash text\{prev\}\}$ through the damped velocity-Verlet step:

second order:  h_new = LN( h + delta/(1+dt*gamma) + dt^2/(m_b*(1+dt*gamma)) * f )
Fock-G1:       h_new = LN( h + beta*f ),   beta = dt^2 / (m_b*(1+dt*gamma_star))

Fock-G1 is produced by overriding a single method (_fock_layer_step) so that h_prev := h is passed down at every layer, making $\delta \equiv 0\backslash delta\; \backslash equiv\; 0$ identically — see model_first_order_fock_g1.py. Every other channel (\(V_\theta\), $V_{\varphi }V\_\backslash phi$, the xi-channel pool, register creation/destruction, the reverse channel, the Fock-coupling regulariser) is inherited unchanged, so the ablation isolates the inertial term only.

Perplexity: a statistical tie

Model	Best val PPL	Step
Fock-G1, first-order (this model)	8.95	18,800
Second-order anchor (gamma*=0.30)	9.04	19,200
$\mathrm{\Delta }={\text{PPL}}_{\text{FO}}-{\text{PPL}}_{\text{anchor}}=-0.09\backslash Delta\; =\; \backslash text\{PPL\}\_\{\backslash text\{FO\}\}\; -\; \backslash text\{PPL\}\_\{\backslash text\{anchor\}\}\; =\; -0.09$, against a pre-registered minimum detectable effect of ${\mathrm{\Delta }}_{\min }=1.00\backslash Delta\_\{\backslash min\}\; =\; 1.00$ PPL for this protocol — a single-seed reading that is H0-consistent: first-order matches second-order on this corpus at this scale. (A full multi-seed replication, as run for the TinyShakespeare comparison discussed below, has not yet been run for this Fock-PARFLM/TinyStories pairing.)

Comparing the shape of the potential, not just the loss

Matching PPL does not by itself imply matching internal structure — two models can reach the same loss through different learned solutions. To test this directly, both checkpoints' $V_{\theta }V\_\backslash theta$ submodules (AnisotropicDepthConditionedGaussianVTheta) were reconstructed from their state dicts alone (no GPU, no TinyStories data required) and probed at 20 jittered samples of each layer's own learned depth code, then summarised by five architecture-level statistics per well: the smallest and largest eigenvalue of ${\mathrm{\Sigma }}_k^{-1}\backslash Sigma\_k^\{-1\}$ (curvature range), the trace (total curvature), the ratio ${\lambda }_{\max }\mathrm{/}{\lambda }_{\min }\backslash lambda\_\{\backslash max\}/\backslash lambda\_\{\backslash min\}$ (anisotropy), the entropy of the mixture weights $w_{k}w\_k$ (how evenly the 8 wells per context are used), and the nearest-neighbour distance between attractor centres (how tightly packed the 32 wells are).

As a control for "how much difference should be considered noise", the same five statistics were also computed between two second-order checkpoints that differ only in the damping coefficient (gamma=0.30 vs. gamma=0.15, both from the anchor's own gamma-sweep) — i.e. a same-order, same-seed, same-architecture baseline for how much these statistics naturally move around under an unrelated hyperparameter change.

Metric (mean)	First-order vs. anchor (gamma*=0.30)	Same-order baseline (anchor gamma=0.30 vs. gamma=0.15)
Smallest curvature eigenvalue (lambda_min)	-4.7%	-5.8%
Largest curvature eigenvalue (lambda_max)	+7.9%	+10.8%
Total curvature (trace)	+5.6%	+19.9%
Anisotropy ratio (lambda_max / lambda_min)	+10.8%	+19.8%
Mixture-weight entropy (fraction of max)	-0.7 pp	-5.3 pp
Nearest-neighbour attractor spacing	+0.8%	+14.6%
Depth-code cosine similarity (raw weights)	0.982	0.930

Every single statistic differs less between first-order and second-order than between two second-order checkpoints trained at different damping. The training-order gap sits comfortably inside the natural hyperparameter noise floor for this architecture, at this scale, on this corpus. Combined with the near-identical learned xi-channel decays (\([0.418, 0.611, 0.792, 0.923]\) vs. $[0.418,0.609,0.788,0.923][0.418,\; 0.609,\; 0.788,\; 0.923]$) and the statistically tied PPL, the honest reading is: for all practical purposes, first-order and second-order training converge to the same $V_{\theta }V\_\backslash theta$ potential on TinyStories at this scale.

(One nuance worth flagging honestly: the raw per-tensor weight cosine similarities for the individual projection layers inside each well bank — mu_proj, a_proj.weight, w_proj, B_proj — are low (0.09-0.53) in both comparisons above, first-order-vs-anchor and same-order-baseline alike. This is expected and uninformative: the mixture has gauge freedom (wells can permute, the rank-4 factor B_k has an orthogonal-rotation ambiguity), so raw weight vectors are not directly comparable across any two independently-initialised-and-trained runs, regardless of integrator order. The one systematic exception is a_proj.bias — the precision-magnitude bias — whose cosine similarity is >0.999 in both comparisons, suggesting it converges to a near-universal value dictated mostly by the precision cap rather than by task-specific structure. The five aggregate statistics in the table above, which are invariant to well permutation and to the specific rotation of B_k, are the meaningful comparison; the raw-weight numbers are reported in fock_g1_vs_second_order_report.json for completeness, not as evidence of dissimilarity.)

This is not a general result — the TinyShakespeare counter-example

A structurally similar question — does second-order training leave a measurable trace beyond what shows up in the loss — was already tested on a smaller, unrelated architecture (SPLM em_ln, not Fock-PARFLM; $d=128d=128$, no Fock registers, no anisotropic Gaussian) trained on TinyShakespeare, using a different, more sensitive methodology: a pre-registered Markov-order regression (kernel-ridge, LOSO, $p=50p=50$) on the hidden-state trajectories at inference time, testing whether the immediate-past state alone predicts the next state as well as the past two states do.

• On raw PPL, that experiment reached the same qualitative conclusion as this one: under a leak-free integrator, the second-order arm (SPLM-2, gamma*=0.30) and its first-order ablation (SPLM-1) were statistically indistinguishable across 3 seeds (\(\bar\Delta = +1.27\) PPL, paired $t=1.00t=1.00$, $p\approx 0.42p\backslash approx0.42$) — see RESULTS_LEAKFREE.md in the companion repo.
• But the Markov-order regression did find a small, highly significant difference the PPL comparison missed: SPLM-1's hidden-state dynamics were indistinguishable from a pure first-order (Markov-1) process (\(\rho_{12} \approx 1.0035\), per-seed $p\in \{0.53,0.77,0.93\}p\; \backslash in\; \backslash \{0.53,\; 0.77,\; 0.93\backslash \}$), while the trained second-order SPLM-2 retained a small but robust "inertial trace" (\(\rho_{12} \approx 1.034\), per-seed $pp$ as small as $1.1\times {10}^{-26}1.1\backslash times10^\{-26\}$; cross-arm paired $t=10.27t=10.27$, $p=0.005p=0.005$) — both arms were "observationally first-order" in the sense the paper's Lemma 9 requires, but the second-order arm was measurably closer to second-order than the first-order arm was, by about 3% per token.

That is a genuine, statistically-supported structural fingerprint of second-order training that a PPL comparison alone would have missed entirely. It is exactly the kind of effect this card's own comparison — a static, architecture-only probe of $V_{\theta }V\_\backslash theta$'s parameter-space geometry — is not designed to detect. The two studies differ in architecture, corpus, and (most importantly) methodology: this card compares frozen potential parameters evaluated at synthetic probe points, while the TinyShakespeare study compares trajectories the trained model actually produces at inference. A like-for-like version of the TinyShakespeare test — the same Markov-order regression, run on this checkpoint's and the anchor's inference-time hidden states on real TinyStories validation data — has not yet been run and is the natural next step before concluding that first-order training is dynamically equivalent here, not merely loss-equivalent and parameter-geometry-equivalent.

Expectation for OpenWebText: probably not this rosy

Earlier depth-conditioned isotropic-Gaussian scale-up runs on richer corpora (the d=384/d=768/d=1024 gamma-sweep lines in this family) showed noticeably more heterogeneous, less well-behaved training dynamics than anything observed at TinyStories scale — larger and more variable force magnitudes, and damping-regime sensitivity that only appears once the corpus and hidden width grow past this toy scale. The theoretical expectation (see the companion paper's discussion of which geometric capabilities survive the overdamped/first-order reduction) is that the inertial term matters more precisely in these higher-curvature, less quasi-static regimes. TinyStories — a small, synthetic, low-perplexity-ceiling corpus capped at 5M training tokens — is close to the gentlest possible test case for "can first-order training match second-order". We do not expect this near-equivalence to hold at OpenWebText scale, where a Fock-G1 ablation has not yet been run; when those results are available, this subsection (and the corresponding discussion in the companion paper) will be updated accordingly.

How to Get Started

import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFConfig
from model_first_order_fock_g1 import FockG1MultiXiPARFLM
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing

# -- Build base model (first-order integrator) --
config = FockMultiXiPARFConfig(
    vocab_size=50257, d=256, max_len=1024,
    L=8, v_hidden=1024, v_depth=3, dt=1.0,
    mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    logfreq_init_alpha=0.1,
    init_gamma=1.0,
    fixed_gamma=0.30,          # only used to parameterise beta; no damping at inference
    causal_force=True,
    ln_after_step=True,
    xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True,
    xi_alpha_init_mode="explicit",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
    fock_version="v2",
    n_registers=16,
    register_salience_decay=0.5,
    register_salience_threshold=0.005,
    creation_gate_hidden=64,
    stack_discipline=True,
    d_k=64,
    tau_create_init=8.0,
    reverse_channel=True,
    per_register_tau=True,
    per_register_keys=True,
    ortho_register_init=True,
    prefix_causal_registers=True,
)
model = FockG1MultiXiPARFLM(config)   # <-- first-order subclass

# -- Swap in the depth-conditioned anisotropic multi-context Gaussian V_theta --
import math
model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=256, K=8, n_ctx=4, n_layers=8, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(256),
    precision_max=2.0 / 256,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-fock-g1",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly (32 wells: 4 contexts x 8 wells) --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                                # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)              # (1, 64, 4, 8, 256)
    print(f"Attractor centres shape: {centres.shape}")

Available Artifacts

File	Description
checkpoint/ckpt_best.pt	Best checkpoint (8.95 PPL at step 18,800), trained natively leak-free
training_log.jsonl	Per-step training metrics, including causal_probe and trained_leak_probe events
training_curve_fock_g1.png	Training/validation loss curves
v_theta_hist_aniso_gaussian.png	V_theta output distribution histogram
landscape_stats_aniso_gaussian.json	V_theta landscape statistics (mean, std, min, max, range)
model_aniso_gaussian_vtheta.py	Anisotropic Gaussian V_theta classes (single-bank, multi-context, depth-conditioned) + install_aniso_depth_routing (identical to the anchor's)
model_first_order_fock_g1.py	The FockG1MultiXiPARFLM subclass implementing the first-order ablation (_fock_layer_step override)
fock_g1_vs_second_order_report.json	Full architecture-level V_theta comparison report (this model vs. the second-order anchor)
same_order_baseline_report.json	Same-order noise-floor baseline report (anchor gamma=0.30 vs. gamma=0.15), used for the comparison in First-Order vs. Second-Order
compare_vtheta_profiles.py	The comparator script used to generate both reports above
config.json	Model configuration

Training Details

Training Data

TinyStories --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens. Identical data pipeline and seed to the second-order anchor.

Training Procedure

The base architecture, $V_{\theta }V\_\backslash theta$ design, $V_{\varphi }V\_\backslash phi$ pairwise force, Fock register pool, and Fock-coupling regulariser are all identical to the second-order anchor. The only structural change is the integrator: FockG1MultiXiPARFLM overrides _fock_layer_step to force the velocity memory to zero at every layer (see First-Order vs. Second-Order). prefix_causal_registers=True is set from the start, as in the anchor.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	5e-4 (cosine decay)
Warmup steps	400
Gradient clipping	1.0 (0.5 for Fock-register gradients, per-group)
Batch size	16 (4 x grad-accum 4)
Block size	512
Training steps	20,000
lambda_V (V_theta regularisation)	0.01
lambda_fock (coupling regulariser)	0.005
Hardware	NVIDIA T1/A100 (Google Colab)

Causal-Leak Verification (no fix needed — trained natively leak-free)

As with the anchor, this run started with prefix_causal_registers=True from step 1, so the training log carries the same probes as routine certification, not remediation:

step	causal_probe_max_delta	causal_probe_passed	honest vs. standard PPL diff (nats)
4,000	0.0	True	—
8,000	0.0	True	+0.0305 ± 0.0225 SE
12,000	0.0	True	—
16,000	0.0	True	+0.0155 ± 0.0203 SE
20,000	0.0	True	—

The deterministic architectural probe is bit-exact 0.0 at every checkpoint, and the honest-vs-standard PPL differences are within ~1.5 standard errors of zero — not statistically distinguishable from no leak. No causal-leak caveat applies to the 8.95 PPL reported on this card.

Training Script

notebooks/conservative_arch/first_order_ablation/colab_fock_g1_aniso_gaussian_fockreg_tinystories.ipynb (companion repo) --- Colab notebook defining FockG1MultiXiPARFLM, running the smoke test that confirms the first- and second-order models diverge from layer 1 onward, then training to 20,000 steps. The architecture-level comparison against the second-order anchor is notebooks/conservative_arch/first_order_ablation/scripts/compare_vtheta_profiles.py.

Evaluation Results

TinyStories Validation Perplexity

Model	PPL	Params	Dynamics order	Bounded V_theta	Causal leak
Matched Attention (baseline)	7.81	19.5M	---	---	No
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, Fock-G1 first-order, this model)	8.95	26.6M	First-order gradient flow	Yes	Never leaked
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, second-order anchor)	9.04	26.6M	Second-order damped Verlet	Yes	Never leaked
Fock-PARFLM v2.1 (MLP)	9.70	17.4M	Second-order damped Verlet	No	Fixed & verified
Fock-PARFLM v2.1 (SQ3)	10.90	18.2M	Second-order damped Verlet	No	Fixed & verified
Fock-PARFLM v2.1 (isotropic Gaussian)	16.33	18.2M	Second-order damped Verlet	Yes	Never leaked

All PPL values in this table are honest, leak-free numbers. This model is the first first-order-trained Fock-PARFLM checkpoint published in this family, and the only one that matches (in fact, marginally beats) its own second-order counterpart; see First-Order vs. Second-Order for why this is a scale/corpus-specific finding, not a general one.

PPL is evaluated on the TinyStories validation split; see Training Data.

V_theta Landscape Statistics

Metric	This model (Fock-G1, first-order)	Second-order anchor	Isotropic Gaussian	Fock-PARFLM (SQ3)
Mean V_theta	-0.0000036	-0.0000024	-0.017	0.017
Std V_theta	0.000168	0.0001	0.024	0.42
Range	0.01654	0.0102	0.624	16.0
Theoretical bound	[-4, 0]	[-4, 0]	[-4, 0]	(-∞, +∞)

The raw output range is ~1.6x wider than the second-order anchor's — a modest, real difference in how far $V_{\theta }V\_\backslash theta$'s scalar value swings on real validation batches — but both remain two orders of magnitude below the theoretical bound and, per the architecture-level comparison in First-Order vs. Second-Order, the underlying curvature/anisotropy/entropy/spacing statistics of the wells producing that output are close enough to fall inside the same-order noise floor.

Learned Xi-Channel Decay Rates

Final learned alphas $[{\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4]=[0.418,0.611,0.792,0.923][\backslash alpha\_1,\; \backslash alpha\_2,\; \backslash alpha\_3,\; \backslash alpha\_4]\; =\; [0.418,\; 0.611,\; 0.792,\; 0.923]$ (initialised at $[0.25,0.5,0.75,0.95][0.25,\; 0.5,\; 0.75,\; 0.95]$) — within 0.002-0.004 of the second-order anchor's $[0.418,0.609,0.788,0.923][0.418,\; 0.609,\; 0.788,\; 0.923]$ at every channel, despite the two models never sharing a single training step. The Fock-coupling regulariser's effect on channel engagement is evidently a property of the loss, not of the integrator.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Model	Design	PPL	HuggingFace
Multi-Xi SPLM (MLP)	Pure scalar potential	11.51	semsimula-splm-multixi
Multi-Xi PARFLM (MLP)	Scalar + pairwise forces	12.06	semsimula-parflm-multixi
Fock-PARFLM v2.1 (MLP)	PARFLM + Fock registers	9.70	semsimula-fock-parflm
Fock-PARFLM v2.1 (SQ3)	Structured + pairwise + Fock	10.90	semsimula-fock-parflm-structured-vtheta
Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian)	Bounded multi-context + pairwise + Fock	16.33	semsimula-fock-parflm-depthcond-vtheta
Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian, second-order anchor)	Bounded, ellipsoidal multi-context + pairwise + Fock	9.04	semsimula-fock-parflm-anisogaussian-vtheta
Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian, first-order, Fock-G1)	Same, first-order gradient-flow integrator	8.95	this model
Fock-PARFLM v2.1 (depth-cond. Gaussian, OpenWebText scale)	Bounded multi-context + pairwise + Fock, d=384 L=16	27.23	semsimula-fock-parflm-depthcond-vtheta-openwebtext
Fock-Attention	Fock + attention	9.42	semsimula-fock-attention
Hybrid SPLM+Attn	Attention + SPLM refinement	8.50	semsimula-hybrid-splm

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Research checkpoint only. This model is a proof-of-concept ablation studying whether first-order training can replace second-order (damped Lagrangian) training in a conservative-by-construction language model, not a production system.
• TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
• English only. No multilingual capability.
• Single seed. Both the PPL comparison and the V_theta shape comparison against the second-order anchor use a single training seed each. A full multi-seed replication (as run for the TinyShakespeare SPLM-1/SPLM-2 comparison discussed in First-Order vs. Second-Order) has not yet been run for this Fock-PARFLM/TinyStories pairing.
• The V_theta shape comparison is architecture-only, not data-conditional. compare_vtheta_profiles.py reconstructs the $V_{\theta }V\_\backslash theta$ submodule from each checkpoint's state dict and probes it at synthetic depth-code-plus-jitter inputs; it does not run the full model on real TinyStories validation data to see the $\xi \backslash xi$ contexts the trained model would actually produce. A data-conditional follow-up is a natural next step.
• A more sensitive dynamical test has not yet been run on this pairing. The TinyShakespeare study found a structural difference between first- and second-order training that a PPL comparison alone missed, using a dedicated hidden-state Markov-order regression at inference time. The equivalent test has not yet been run on this Fock-PARFLM/TinyStories checkpoint pair; see First-Order vs. Second-Order.
• Not yet validated at scale. This near-equivalence between first- and second-order training is expected to be corpus- and scale-dependent, and is explicitly not expected to hold at OpenWebText scale, where a Fock-G1 ablation has not yet been run; see Expectation for OpenWebText.
• No safety training. No RLHF, DPO, or safety filtering has been applied.
• V_phi and Fock forces still use autograd. Only the $V_{\theta }V\_\backslash theta$ gradient is analytical; the $V_{\varphi }V\_\backslash phi$ pairwise force and Fock register forces still require torch.autograd.grad.
• No causal-leak issue. Like the anchor, this checkpoint was trained natively with prefix_causal_registers=True and needed no re-training or PPL correction; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: NVIDIA T4/A100 (Google Colab, resumed across sessions)
• Training time: 20,000 steps
• Carbon footprint: Estimated less than 3 kg CO2