dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep

Fock-PARFLM v2.1 Anisotropic Gaussian V_theta + Fock Regularisation — Gamma Sweep with Geodesic Residual Analysis (OpenWebText, d=1024)

This repository holds eight short (3,000-step) training runs, one per candidate damping coefficient $\gamma \in \{0.05,0.10,0.15,0.20,0.25,0.30,0.40,0.50\}\backslash gamma\; \backslash in\; \backslash \{0.05,\; 0.10,\; 0.15,\; 0.20,\; 0.25,\; 0.30,\; 0.40,\; 0.50\backslash \}$, of the depth-conditioned anisotropic Gaussian $V_{\theta }V\_\backslash theta$ with Fock-coupling regularisation architecture, scaled up to d=1024, L=16 and trained on OpenWebText. This is not a final trained model — it is the diagnostic sweep used to pick the damping coefficient for a subsequent full 100,000-step training run at this scale. Each of the 8 checkpoints is included in full, and (unlike the isotropic-Gaussian d=1024 sweep elsewhere in this family, which only completed 4/8 candidates) all 8 candidates here completed successfully.

Alongside the perplexity sweep, every checkpoint is also scored with the damped-geodesic residual $\bar{R}(\gamma )\backslash bar\{R\}(\backslash gamma)$ — a closed-form diagnostic (no additional training, no autodiff through a learned metric) that measures how closely each model's own hidden-state trajectory follows a geodesic of the Riemannian (Jacobi) metric induced by its own learned potential. The headline result:

Both the perplexity minimum and the geodesic-residual minimum land at $\gamma =0.050\backslash gamma\; =\; 0.050$ — the smallest candidate tested, the same boundary optimum shape found at d=768, but here with the widest margin of any boundary-optimum sweep in the family (~10.5% over the runner-up) and, unlike d=768, a geodesic-residual curve that rises essentially monotonically past the minimum. This is also an exact, zero-parameter match to the two-regime closed-form damping predictor's high-\(d\) anchor prediction — the second consecutive exact confirmation at $d\ge 768d\; \backslash geq\; 768$ after d=768, and the third overall counting the original MLP architecture's own d=1024 result. See Coincidence, Boundary, and a Second Exact Predictor Match.

Based on this sweep, a full 100,000-step run at $\gamma =0.05\backslash gamma\; =\; 0.05$ is expected to follow the same launch pattern as the d=384 and (pending) d=768 runs; as of this card's creation the d=1024 full run has not yet been launched.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Repository
• Architecture
• The Gamma Sweep
• The Geodesic Residual Diagnostic
• Results: The Widest Boundary-Optimum Margin Yet
• The PPL Wiggle Without the Geodesic Wiggle
• Coincidence, Boundary, and a Second Exact Predictor Match
• Comparison Across All Three Widths
• Caveats: Short-Sweep Reliability and Two Incomplete Training Logs
• How to Get Started
• Available Artifacts
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

When to Use This Repository

Use this repository if you want to:

• Reproduce or extend the gamma-selection methodology for the anisotropic-Gaussian + Fock-reg Fock-PARFLM line at d=1024, L=16 on OpenWebText, including the geodesic-residual diagnostic.
• Study the PPL-only non-monotonic wiggle: at this width the PPL-vs-gamma curve wiggles (as at d=768) but the geodesic-residual curve does not — see below.
• Compare geodesic fidelity across damping regimes on a bounded, analytically-differentiable potential — the closed-form Jacobi-metric machinery here is structurally unavailable to attention-based or MLP-potential architectures.
• Study the widest and cleanest boundary optimum in the family, useful as the "textbook case" for what a reliable short-sweep result looks like, in contrast to the flat-bowl / reversal-prone cases at $d\le 384d\; \backslash leq\; 384$.

Do not use this repository if you want a well-trained OpenWebText language model: every checkpoint here has seen only 3,000 steps (~50M tokens at effective batch 16 x block 512) and none is intended to produce fluent text. For a fully trained OpenWebText-scale Fock-PARFLM checkpoint, see semsimula-fock-parflm-depthcond-vtheta-openwebtext (27.23 PPL, isotropic Gaussian, d=384, 250K steps) — no full-length d=1024 run in this line exists yet.

Architecture

Identical Fock-PARFLM v2.1 scaffold to the d=384 and d=768 gamma-sweep siblings, scaled up to d=1024, with the same d768-style force-bounding for numerical stability plus a further-lowered peak learning rate:

Input tokens x_1, ..., x_T
       |
   Untied token embedding E[x] + learned positional P[t]
       |
   For each of L=16 damped-Euler integration steps (shared force field):
       |
       +-- K=5 causal-EMA context channels:
       |     xi^(m)_t = causal_ema(h, alpha_m)      [horizons ~2 .. ~200 tokens]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian, force-bounded):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [5 contexts, 40 wells total]
       |     f_theta = clamp(-analytical_grad_h V_theta, max_norm=2/sqrt(1024))
       |
       +-- Sparse pairwise V_phi (structural-competitive, 4 heads):
       |     top-k=16 past tokens per query (Gumbel routing)
       |     f_phi = -grad_h V_phi(h_t, h_s)                [autograd, sparse]
       |
       +-- Fock register pool (v2, 32 registers):
       |     M=32 virtual registers, Q/K/V creation gates, d_k=64
       |     LIFO stack discipline, per-register tau/keys, ortho init
       |     register repulsion (Gram penalty, lambda=0.05)
       |     reverse channel (per-layer, stabilised, pre-LN, soft-norm, warmup 4000)
       |     prefix-causal (leak-free by construction)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = clamp(f_theta + f_phi + f_fock, max_norm=2/sqrt(1024))
       |
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       |
       +-- LayerNorm(h)
       |
   Logits = h @ W_out^T + b_out                              [UNTIED W_out]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

Parameter	Value
Hidden dim (d)	1024
Layers (L)	16
Max sequence length	1024 (trained at block 512)
Vocab (GPT-2 BPE)	50257
V_theta kind	Depth-conditioned multi-context anisotropic Gaussian (bounded mixture)
V_theta contexts (n_ctx)	5 (one bank per xi channel)
Wells per context (K)	8
Total attractors	40
Anisotropic rank (r)	4 — low-rank factor B_k in R^(1024x4) per well
Precision init / cap	a_k init at -log(1024) (log-precision), capped at 2/1024
Force bounding	$V_{\theta }V\_\backslash theta$'s own gradient norm and the model-level total force are both clamped to $2\mathrm{/}\sqrt{1024}=0.06252/\backslash sqrt\{1024\}\; =\; 0.0625$ (same style introduced at d=768; not present in the d=384 sweep)
Depth codes	per-layer additive shift, shape (L=16, n_ctx=5, d=1024), init std 0.02
Xi channels (K_xi)	5 (alpha inits 0.50, 0.75, 0.95, 0.99, 0.995)
Fock-coupling regulariser	log-barrier on alpha_k, lambda=0.005, eps=1e-6 (see the TinyStories anchor for the full derivation)
V_phi kind	structural_competitive
V_phi heads / d_type / d_angle	4 / 32 / 16
Top-k (sparse routing)	16
Fock version	v2
Registers (M)	32
Register d_k	64
Stack discipline	LIFO
Reverse channel	Yes — per-layer, stabilised, pre-LN, soft-norm, 4,000-step warmup
Register repulsion	Gram penalty, lambda=0.05
Embeddings	Untied (separate W_out)
Mass model	logfreq (frozen OpenWebText surprisal lookup)
lambda_V (V_theta regularisation)	0.01
Prefix-causal registers	Yes — every checkpoint trained natively leak-free from step 0
Total parameters	363,884,315
V_theta parameters	252,026,920 (69% of total — the largest $V_{\theta }V\_\backslash theta$ share yet in this family; the low-rank correction $B_k\in {\mathbb{R}}^{d\times r}B\_k\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; r\}$ scales with $dd$, so widening from d=768 to d=1024 pushes $V_{\theta }V\_\backslash theta$'s share from 63% to 69%)

The analytical form of $V_{\theta }V\_\backslash theta$ — the diagonal-plus-rank-4 precision ${\mathrm{\Sigma }}_k^{-1}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(a_k)+B_k{B}_k^{\mathrm{\top }}\backslash Sigma\_k^\{-1\}\; =\; \backslash mathrm\{diag\}(a\_k)\; +\; B\_k\; B\_k^\backslash top$, the depth-conditioning mechanism, and the closed-form bounded gradient — is unchanged from the TinyStories anchor and the d=384/d=768 siblings; only $dd$ changes, plus the force-norm clamp inherited from d=768.

The Gamma Sweep

Eight candidate damping coefficients $\gamma \in \{0.05,0.10,0.15,0.20,0.25,0.30,0.40,0.50\}\backslash gamma\; \backslash in\; \backslash \{0.05,\; 0.10,\; 0.15,\; 0.20,\; 0.25,\; 0.30,\; 0.40,\; 0.50\backslash \}$ were each trained from scratch for 3,000 steps (WSD schedule, peak LR 1.5e-4 — half of the d384/d768 sweeps' 3e-4, a d1024-specific stability adjustment — effective batch 16, up to 1B-token training pool), then scored on a held-out 2M-token OpenWebText validation slice. This protocol mirrors the d=384 and d=768 sweeps and is designed to be cheap: 8 x 3,000 steps rather than 8 x 100,000. Unlike d=768's batch=2 x accum=4, this sweep reverts to batch=1 x accum=16 (effective 16, matching d=384) with the wider global/V_phi gradient-clip ceilings (1.0/0.3) also matching d=384 rather than d=768's tighter 0.5/0.2 — the halved peak LR was evidently sufficient stabilisation at this width on its own.

The Geodesic Residual Diagnostic

Fock-PARFLM's scalar potential $V_{\theta }V\_\backslash theta$ is a closed-form Gaussian mixture with an analytical gradient, which makes a diagnostic available here that is structurally unavailable to attention-based or MLP-potential architectures: at fixed energy $EE$, Hamiltonian trajectories are geodesics of the Jacobi metric $g_{ij}^J(x)=2(E-V(x)){\delta }_{ij}g^J\_\{ij\}(x)\; =\; 2(E\; -\; V(x))\backslash delta\_\{ij\}$, a conformally flat metric whose Christoffel symbols are closed-form functions of $\mathrm{\nabla }{V}_{\theta }\backslash nabla\; V\_\backslash theta$ — no learned metric, no autodiff through a metric, no boundary-value solve.

For a trajectory with position stream $x_{\mathrm{\ell }}x\_\backslash ell$, velocity stream $v_{\mathrm{\ell }}v\_\backslash ell$, and measured acceleration $a_{\mathrm{\ell }}a\_\backslash ell$ (the discrete second difference of $x_{\mathrm{\ell }}x\_\backslash ell$, consistent with the model's Velocity-Verlet-style integrator), the per-layer damped-geodesic residual is

$$R_{\mathrm{\ell }}=\frac{\parallel a_{\mathrm{\ell }}+\mathrm{\Gamma }(v_{\mathrm{\ell }},v_{\mathrm{\ell }})+\gamma v_{\mathrm{\ell }}\parallel }{\parallel a_{\mathrm{\ell }}\parallel +\epsilon },\mathrm{\Gamma }(v,v{)}^k={\mathrm{\Gamma }}_{ij}^k{v}^i{v}^j,R\_\backslash ell\; =\; \backslash frac\{\backslash big\backslash lVert\; a\_\backslash ell\; +\; \backslash Gamma(v\_\backslash ell,\; v\_\backslash ell)\; +\; \backslash gamma\; v\_\backslash ell\; \backslash big\backslash rVert\}\{\backslash lVert\; a\_\backslash ell\; \backslash rVert\; +\; \backslash varepsilon\},\; \backslash qquad\; \backslash Gamma(v,v)^k\; =\; \backslash Gamma^k\_\{ij\}\; v^i\; v^j,$$

where $\mathrm{\Gamma }\backslash Gamma$ is computed in closed form from $V_{\theta }V\_\backslash theta$'s analytical gradient. $R_{\mathrm{\ell }}\approx 0R\_\backslash ell\; \backslash approx\; 0$ means the trajectory is a damped geodesic of the metric induced by the model's own learned potential — this is not a pure-conservation claim (the explicit $\gamma v_{\mathrm{\ell }}\backslash gamma\; v\_\backslash ell$ damping term is included), only that the dynamics satisfy the damped geodesic equation with the architecture's own damping coefficient. Averaging over layers and 10 fixed validation batches (seed 42) gives $\bar{R}({\gamma }_{\text{train}})\backslash bar\{R\}(\backslash gamma\_\{\backslash text\{train\}\})$, evaluated at ${\gamma }_{\text{eval}}={\gamma }_{\text{train}}\backslash gamma\_\{\backslash text\{eval\}\}\; =\; \backslash gamma\_\{\backslash text\{train\}\}$ for each retained checkpoint — the diagonal overlay against $\mathrm{P}\mathrm{P}\mathrm{L}({\gamma }_{\text{train}})\backslash mathrm\{PPL\}(\backslash gamma\_\{\backslash text\{train\}\})$.

A second, closed-form quantity — the recovered intrinsic damping ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ — is the least-squares row minimiser "the damping value that best explains this specific trajectory," independent of what $\gamma \backslash gamma$ the model was actually trained with:

$${\gamma }_{\text{geo}}=-\frac{⟨a_{\mathrm{\ell }}+\mathrm{\Gamma }(v_{\mathrm{\ell }},v_{\mathrm{\ell }}),v_{\mathrm{\ell }}⟩}{\parallel v_{\mathrm{\ell }}{\parallel }^2}\mathrm{.}\backslash gamma\_\{\backslash text\{geo\}\}\; =\; -\backslash frac\{\backslash big\backslash langle\; a\_\backslash ell\; +\; \backslash Gamma(v\_\backslash ell,\; v\_\backslash ell),\backslash \; v\_\backslash ell\; \backslash big\backslash rangle\}\{\backslash lVert\; v\_\backslash ell\; \backslash rVert^\{2\}\}.$$

Full derivation, practical mitigations (turning-point exclusion, reference-energy convention, integrator staggering), and validation controls (vanilla-baseline, shuffled-\(\Gamma\), and random-direction nulls) are in the companion note Geodesic_Preservation_Experiment.md.

Results: The Widest Boundary-Optimum Margin Yet

$\gamma \backslash gamma$	PPL	$\bar{R}\backslash bar\{R\}$	${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$	Excluded frac
0.050	244.23 ← PPL min	1.2066 ← $\bar{R}\backslash bar\{R\}$ min	0.9580	0%
0.100	270.80	1.3139	0.9690	0%
0.150	264.77	1.4842	0.9628	0%
0.200	288.50	1.5201	0.9643	0%
0.250	284.46	1.5358	0.9632	0%
0.300	295.05	1.8097	0.9622	0%
0.400	298.05	2.0380	0.9613	0%
0.500	269.77	2.2954	0.9606	0%

The minima coincide, at the boundary of the swept range, with the widest margin yet. Both PPL and $\bar{R}\backslash bar\{R\}$ bottom out at $\gamma =0.050\backslash gamma\; =\; 0.050$; $\gamma =0.05\backslash gamma=0.05$ beats its nearest PPL competitors by $\approx 10\text{-}11\mathrm{\%}\backslash approx10\backslash text\{-\}11\backslash \%$ (25.5-26.6 PPL against $\gamma =0.50\backslash gamma=0.50$ and $\gamma =0.10\backslash gamma=0.10$ respectively) — comfortably wider than the d=768 sweep's $\approx 4\mathrm{\%}\backslash approx4\backslash \%$ margin and well clear of the family's informal $\approx 5\mathrm{\%}\backslash approx5\backslash \%$ "unreliable ranking" threshold. ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ again clusters tightly: mean $0.96270.9627$, std $0.00300.0030$ — essentially flat across the range, centred about $0.020.02$ lower than the d=768 mean (\(0.9810\)), a small but consistent downward drift with increasing $dd$ that also appears at the isotropic-Gaussian d=1024 point elsewhere in this family (\(\gamma_{\text{geo}} \approx 0.933\)).

The per-layer residual heatmap below shows the departures from geodesic behaviour concentrating in the middle layers as $\gamma \backslash gamma$ increases, peaking sharply around layers 4-7 at $\gamma =0.50\backslash gamma=0.50$:

Unlike d=768, the recovered intrinsic damping here is not just flat but forms the cleanest, tightest line of any sweep in the family:

The PPL Wiggle Without the Geodesic Wiggle

The PPL curve is not monotonic past its minimum — it rises from 244.23 (\(\gamma=0.05\)) to a peak at $\gamma =0.40\backslash gamma=0.40$ (298.05), then drops back down at $\gamma =0.50\backslash gamma=0.50$ (269.77) to nearly the level of $\gamma =0.15\backslash gamma=0.15$ (264.77). This is the same shape found in the d=768 sweep (a local peak in the $\gamma \approx 0.20\text{-}0.40\backslash gamma\; \backslash approx\; 0.20\backslash text\{-\}0.40$ range, partial recovery at 0.40-0.50) — the same shape appearing at two different widths is a weak argument against pure single-run noise as the sole explanation.

What is new at this width: the geodesic-residual curve $\bar{R}(\gamma )\backslash bar\{R\}(\backslash gamma)$ does not share the wiggle. $\bar{R}\backslash bar\{R\}$ rises essentially monotonically across the entire swept range (\(1.21 \to 1.31 \to 1.48 \to 1.52 \to 1.54 \to 1.81 \to 2.04 \to 2.30\)) — the cleanest $\bar{R}\backslash bar\{R\}$-vs-\(\gamma\) curve of any sweep in this document, MLP or Gaussian, with no local peak-and-recovery pattern at all. This decouples the two diagnostics in an informative way: whatever is producing the mid-range PPL bump at $\gamma \approx 0.20\text{-}0.40\backslash gamma\; \backslash approx\; 0.20\backslash text\{-\}0.40$ (single-seed optimisation noise, or a structural interaction between the anisotropic precision matrices and mid-range explicit friction) is not simultaneously producing a mid-range departure from geodesic behaviour — the dynamics stay increasingly non-geodesic in a smooth, monotonic way even where the loss itself wiggles. Disentangling the PPL wiggle's cause would require a 2-3 seed rerun of one or two candidates (e.g. $\gamma =0.30\backslash gamma=0.30$-\(0.40\)) at one or both widths — not yet done. See Determining_optimal_gamma_for_Fock-PARFLM.md §14.5 for the cross-scale discussion (written before this sweep's $\bar{R}\backslash bar\{R\}$-vs-PPL decoupling was noted).

Coincidence, Boundary, and a Second Exact Predictor Match

The two-regime closed-form damping predictor, evaluated at $L=16L=16$, $\bar{m}=1.4\backslash bar\{m\}=1.4$ using the high-\(d\) regime constant ${\rho }_{\text{hi}}=0.565\backslash rho\_\{\backslash text\{hi\}\}\; =\; 0.565$, gives the same prediction regardless of $dd$ within the high-\(d\) regime (the formula depends on $LL$ and $\bar{m}\backslash bar\{m\}$, not $dd$ directly):

$${\gamma }_{\text{pred}}^{\ast }=\frac{1.4}{16}\ln (1\mathrm{/}0.565)=0.0875\times 0.571=0.050\backslash gamma^\{\backslash ast\}\_\{\backslash text\{pred\}\}\; =\; \backslash frac\{1.4\}\{16\}\backslash ln(1/0.565)\; =\; 0.0875\; \backslash times\; 0.571\; =\; 0.050$$

Empirical ${\gamma }^{\ast }=0.050\backslash gamma^\{\backslash ast\}\; =\; 0.050$. Exact match, zero error. This is the second consecutive exact match at $L=16L=16$ in the high-\(d\) regime with the aniso-Gaussian $V_{\theta }V\_\backslash theta$ (after the d=768 sweep), and the third exact match overall counting the original MLP-\(V_\theta\) d=1024, L=16 result. Three independent architectures/widths, one unmodified constant (\(\rho_{\text{hi}}=0.565\)), zero free parameters per prediction — the strongest evidence yet that ${\rho }_{\text{hi}}=0.565\backslash rho\_\{\backslash text\{hi\}\}=0.565$ is a genuine architecture-family invariant at $d\gtrsim 768d\; \backslash gtrsim\; 768$.

Sweep	$dd$	$LL$	$V_{\theta }V\_\backslash theta$	${\gamma }_{\text{pred}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash text\{pred\}\}$	${\gamma }_{\text{empirical}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash text\{empirical\}\}$	Match
Original MLP d=1024 sweep	1024	16	MLP	0.050	0.05	Exact
d=768 aniso-Gaussian sweep	768	16	aniso-Gaussian + fock-reg	0.050	0.050	Exact
This sweep	1024	16	aniso-Gaussian + fock-reg	0.050	0.050	Exact

This also matches the winner of the original isotropic-Gaussian d=1024 partial sweep elsewhere in this family (which only completed 4/8 candidates) — giving three separate $V_{\theta }V\_\backslash theta$ variants at this width all agreeing on $\gamma =0.05\backslash gamma=0.05$.

Comparison Across All Three Widths

Sweep	${\gamma }_{\text{PPL}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash text\{PPL\}\}$	${\gamma }_{\bar{R}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash bar\{R\}\}$	Coincidence?	Optimum shape	Margin vs. runner-up	$\bar{R}\backslash bar\{R\}$ curve shape
d=384, L=16	0.100	0.100	Yes — gap 0	Interior minimum	large	non-monotonic (\(\gamma=0.20\) outlier)
d=768, L=16	0.050	0.050	Yes — gap 0	Boundary minimum	~4%, modest	non-monotonic wiggle
d=1024, L=16 (this sweep)	0.050	0.050	Yes — gap 0	Boundary minimum	~10.5%, widest yet	monotonic — cleanest in family

All three widths show coinciding PPL and geodesic-residual minima, but the pattern strengthens with scale: the margin widens (large at d=384 down to a modest 4% at d=768, then back up to a clean 10.5% at d=1024, this time without the ambiguity of an interior-bowl competing basin), and the geodesic-residual curve gets progressively cleaner — from an outlier-marred non-monotonic curve at d=384, to a genuine-but-unexplained wiggle at d=768, to a fully monotonic curve at d=1024. ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ clustering also tightens with scale in absolute spread (though drifting slightly downward in mean, $0.981\to 0.9630.981\; \backslash to\; 0.963$ from d=768 to d=1024) — consistent with a model whose intrinsic damping preference becomes both more uniform and slightly lower as width increases.

Caveats: Short-Sweep Reliability and Two Incomplete Training Logs

This is a 3,000-step, single-seed measurement per gamma, and the family has one documented case where a short-sweep ranking reversed at full training length: the d=256 TinyStories aniso-Gaussian sweep favoured $\gamma =0.150\backslash gamma=0.150$ at 3K steps, but the full 20K-step run favoured $\gamma =0.300\backslash gamma=0.300$ instead. Two considerations favour trusting this sweep's $\gamma =0.05\backslash gamma=0.05$ recommendation more than that precedent would suggest: (a) this is a boundary optimum with ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ essentially flat across the whole range — there is no competing interior basin to misrank within; and (b) the margin (~10.5%) is the widest of any boundary-optimum sweep in the family, and the predictor's independent, zero-parameter prediction lands on the exact same value.

Two data-quality notes specific to this sweep's raw artifacts:

• Two training logs are incomplete. checkpoints/gamma_0.100/training_log.jsonl contains only the final (step-3,000) evaluation record, and checkpoints/gamma_0.200/training_log.jsonl contains only the final two (step-2,500 and step-3,000) records — both missing the earlier per-500-step entries and the step-2,000 causal_probe event that every other candidate's log contains. This looks like a Colab logging/session hiccup specific to those two candidates' runs rather than a training failure: both final checkpoints' val_ppl match sweep_summary.json and geodesic_results.json exactly, and the geodesic residual analysis (which reads directly from the checkpoint, not the training log) completed normally for both. See Causal-Leak Verification below.
• Unlike d=768, this sweep does not show the same non-monotonic pattern in $\bar{R}\backslash bar\{R\}$ (see above) — only in PPL — so the usual "watch for the wiggle's instability signature" caveat applies specifically to the loss curve, not the geometric diagnostic, for a full run at this width.

Recommendation: $\gamma =0.05\backslash gamma=0.05$ for the d=1024, L=16 aniso-Gaussian full run — the highest-confidence recommendation of the three widths swept so far in this line.

How to Get Started

import math, torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing
from huggingface_hub import hf_hub_download

REPO = "dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep"
GAMMA = "0.050"  # the recommended candidate; also available: 0.100, 0.150, 0.200, 0.250, 0.300, 0.400, 0.500

logfreq_path = hf_hub_download(repo_id=REPO, filename="results/logfreq_surprisal_openwebtext.npy")
FORCE_MAX = 2.0 / math.sqrt(1024)

config = FockMultiXiPARFConfig(
    vocab_size=50257, d=1024, max_len=1024, L=16,
    v_hidden=1024, v_depth=3, dt=1.0,
    mass_mode="logfreq", logfreq_path=logfreq_path, logfreq_init_alpha=0.1,
    init_gamma=1.0, fixed_gamma=float(GAMMA),
    causal_force=True, ln_after_step=True,
    xi_channels=5, xi_alpha_inits=[0.50, 0.75, 0.95, 0.99, 0.995],
    xi_learnable=True, xi_alpha_init_mode="explicit",
    fock_version="v2", n_registers=32,
    reverse_channel=True, reverse_channel_stable=True, reverse_channel_pre_ln=True,
    reverse_channel_soft_norm=True, reverse_channel_warmup_steps=4000, reverse_channel_per_layer=True,
    register_repulsion=True, register_repulsion_coeff=0.05,
    prefix_causal_registers=True,
    v_phi_kind="structural_competitive", v_phi_n_heads=4, v_phi_d_type=32, v_phi_d_angle=16,
    v_phi_eps=0.1, v_phi_phi_hidden=128, v_phi_theta_hidden=128, v_phi_mlp_hidden=128,
    top_k=16,
    use_output_bias=True, tie_embeddings=False,
    score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3, gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True, per_layer_v_phi_scale=True,
    register_salience_decay=0.5, register_salience_threshold=0.005,
    creation_gate_hidden=64, stack_discipline=True,
    d_k=64, tau_create_init=8.0,
    per_register_tau=True, per_register_keys=True,
    ortho_register_init=True,
    force_clamp_max=FORCE_MAX,
)
model = FockMultiXiPARFLM(config)

model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=1024, K=8, n_ctx=5, n_layers=16, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(1024),
    precision_max=2.0 / 1024,
    force_norm_max=FORCE_MAX,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)

ckpt_path = hf_hub_download(repo_id=REPO, filename=f"checkpoints/gamma_{GAMMA}/ckpt_best.pt")
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")            # 363,884,315
print(f"gamma={state['gamma']}  step={state['step']:,}  val_ppl={state['val_ppl']:.2f}")

Available Artifacts

Path	Description
checkpoints/gamma_0.050/ckpt_best.pt ... checkpoints/gamma_0.500/ckpt_best.pt	All 8 gamma-sweep checkpoints (step 3,000 each), each a dict with model_state_dict, optimizer_state_dict, step, val_loss, val_ppl, gamma, v_theta_variant, aniso_rank, lambda_fock_reg
checkpoints/gamma_*/training_log.jsonl	Per-500-step training metrics and the step-2,000 causal_probe event, one file per gamma (see caveats — gamma=0.100 and gamma=0.200 logs are truncated)
sweep_summary.json	Sweep-level summary (best PPL per gamma, wall-clock)
results/geodesic_results.json	Full per-gamma geodesic residual data: $\bar{R}\backslash bar\{R\}$, ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$, excluded fraction, and all 15 per-layer residuals
results/geodesic_overlay_aniso_gaussian_d1024.png	PPL vs. geodesic residual dual-axis overlay
results/geodesic_per_layer_aniso_gaussian_d1024.png	Per-layer, per-gamma geodesic residual heatmap
results/gamma_geo_recovery_aniso_gaussian_d1024.png	Recovered intrinsic damping ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ vs. training gamma
results/logfreq_surprisal_openwebtext.npy	Frozen per-token log-frequency (surprisal) lookup used by the logfreq mass model
model_aniso_gaussian_vtheta.py	Anisotropic Gaussian V_theta classes + install_aniso_depth_routing (identical file to the TinyStories anchor's and the d=384/d=768 siblings')
config.json	Full sweep configuration, per-gamma results table, and predictor comparison

Training Details

Training Data

OpenWebText, tokenized with GPT-2 BPE (vocab 50257). Each candidate trains on up to 1B tokens (early stopped at 3,000 steps, effective batch 16, block 512 -> ~24.6M tokens actually consumed) and is evaluated on a held-out 2M-token validation slice with no train/val overlap.

Training Procedure (per gamma candidate)

Hyperparameter	Value
Optimizer	AdamW (betas 0.9/0.95, weight decay 0.01)
LR schedule	WSD (warmup 0-150, stable 150-1950, decay 1950-3000)
Peak learning rate	1.5e-4 (half of d384/d768's 3e-4)
Batch size	1 x grad-accum 16 (effective 16)
Block size	512
Steps	3,000
Gradient clipping	per-group (global 1.0; V_phi 0.3 — same as d=384, wider than d=768's 0.5/0.2)
Force clamp (model-level and V_theta-level)	$2\mathrm{/}\sqrt{1024}=0.06252/\backslash sqrt\{1024\}\; =\; 0.0625$
lambda_V (V_theta regularisation)	0.01
lambda_fock (coupling regulariser)	0.005
Seed	0
Hardware	1x NVIDIA H100/A100 (Google Colab)

Causal-Leak Verification

All 8 checkpoints were trained natively with prefix_causal_registers=True from step 0. The bit-exact future-perturbation causal probe passed with max_delta=0.0 at step 2,000 for 6 of the 8 gamma candidates with an explicit training-log record. The gamma=0.100 and gamma=0.200 training logs are truncated (see caveats) and do not contain this record, but their final checkpoints match the sweep summary and geodesic-results files exactly, and the architecture and training script are identical to the other 6 candidates that did pass explicitly.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_gamma_sweep_geodesic_aniso_gaussian_fockreg_d1024.ipynb (companion repo) — self-contained Colab notebook that runs the 8-candidate sweep, the geodesic residual analysis, and produces the overlay/heatmap/recovery figures in one pass, requiring no additional training beyond the sweep itself.

Evaluation Results

OpenWebText Validation Perplexity (3,000-step sweep candidates)

$\gamma \backslash gamma$	PPL	Rank
0.050	244.23	1st
0.150	264.77	2nd
0.500	269.77	3rd
0.100	270.80	4th
0.250	284.46	5th
0.200	288.50	6th
0.300	295.05	7th
0.400	298.05	8th

These PPL values are from 3,000-step short-sweep candidates and are not comparable to the fully trained OpenWebText checkpoints elsewhere in this family (e.g. 27.23 PPL after 250K steps for the isotropic d=384 flagship). They exist solely to rank candidate damping coefficients. Note the ranking is not simply increasing with $\gamma \backslash gamma$; see The PPL Wiggle Without the Geodesic Wiggle.

See Results: The Widest Boundary-Optimum Margin Yet for the combined PPL / geodesic-residual table.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Model	Design	Corpus	PPL	HuggingFace
Multi-Xi SPLM (MLP)	Pure scalar potential	TinyStories	11.51	semsimula-splm-multixi
Multi-Xi PARFLM (MLP)	Scalar + pairwise forces	TinyStories	12.06	semsimula-parflm-multixi
Fock-PARFLM v2.1 (MLP)	PARFLM + Fock registers	TinyStories	9.70	semsimula-fock-parflm
Fock-PARFLM v2.1 (SQ3)	Structured + pairwise + Fock	TinyStories	10.90	semsimula-fock-parflm-structured-vtheta
Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian)	Bounded multi-context + pairwise + Fock	TinyStories	16.33	semsimula-fock-parflm-depthcond-vtheta
Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian + fock-reg)	Bounded, ellipsoidal multi-context + pairwise + Fock	TinyStories	9.04	semsimula-fock-parflm-anisogaussian-vtheta
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, first-order/Fock-G1)	Same as above, gradient-flow integrator	TinyStories	8.95	semsimula-fock-parflm-anisogaussian-vtheta-fock-g1
Fock-PARFLM v2.1 (depth-cond. Gaussian, OpenWebText scale)	Bounded multi-context + pairwise + Fock, d=384 L=16	OpenWebText	27.23	semsimula-fock-parflm-depthcond-vtheta-openwebtext
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=384)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	278.27 (best of 8, 3K-step sweep, not a final model)	semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=768)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	326.97 (best of 8, 3K-step sweep, not a final model)	semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=1024)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	244.23 (best of 8, 3K-step sweep, not a final model)	this repository
Fock-Attention	Fock + attention	TinyStories	9.42	semsimula-fock-attention
Hybrid SPLM+Attn	Attention + SPLM refinement	TinyStories	8.50	semsimula-hybrid-splm

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Not a final trained model. Every checkpoint in this repository has been trained for only 3,000 steps — a gamma-selection diagnostic, not a language model intended for generation or downstream use. Do not compare its PPL to fully trained checkpoints elsewhere in this family.
• Single-seed sweep. Each gamma candidate is one run; no seed-variance estimate is available. The margin between $\gamma =0.05\backslash gamma=0.05$ and its nearest competitors (~10-11%) is the widest of any boundary-optimum sweep in this family, mitigating (but not eliminating) this risk; see Caveats.
• Two training logs are incomplete (gamma=0.100, gamma=0.200 — missing most per-step entries and the step-2,000 causal-probe record), most likely a Colab logging/session hiccup specific to those runs; the final checkpoints for both are verified consistent with the sweep-summary and geodesic-results files.
• PPL-vs-gamma wiggle, shared with the d=768 companion sweep, is not yet explained (single-seed noise vs. a structural property of this $V_{\theta }V\_\backslash theta$ configuration) — see The PPL Wiggle Without the Geodesic Wiggle. Notably the geodesic-residual curve does not share this wiggle at this width.
• OpenWebText only, English only. No instruction tuning, no RLHF/DPO, no safety filtering.
• Geodesic residual is a diagnostic, not a training objective. The model was trained to minimise cross-entropy; the near-geodesic behaviour at $\gamma =0.05\backslash gamma=0.05$ is an emergent structural property, not something the loss function directly optimises for.
• Very large $V_{\theta }V\_\backslash theta$ hypernetwork. At 252.0M parameters, $V_{\theta }V\_\backslash theta$ alone is 69% of the 363.9M-parameter total — the largest $V_{\theta }V\_\backslash theta$ share of any model in this family, driven by the low-rank correction $B_k\in {\mathbb{R}}^{d\times r}B\_k\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; r\}$ scaling with $dd$.
• No causal-leak issue. All 8 checkpoints were trained natively with prefix_causal_registers=True; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Companion code repository:
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: 1x NVIDIA H100/A100 80GB (Google Colab)
• Training: 8 x 3,000 steps = 24,000 total training steps across the sweep, plus inference-only geodesic residual analysis (10 validation batches x 8 checkpoints, no additional training)
• Carbon footprint: small; a single-GPU research sweep, estimated on the order of a few kg CO2