dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep

Fock-PARFLM v2.1 Anisotropic Gaussian V_theta + Fock Regularisation — Gamma Sweep with Geodesic Residual Analysis (OpenWebText, d=768)

This repository holds eight short (3,000-step) training runs, one per candidate damping coefficient $\gamma \in \{0.05,0.10,0.15,0.20,0.25,0.30,0.40,0.50\}\backslash gamma\; \backslash in\; \backslash \{0.05,\; 0.10,\; 0.15,\; 0.20,\; 0.25,\; 0.30,\; 0.40,\; 0.50\backslash \}$, of the depth-conditioned anisotropic Gaussian $V_{\theta }V\_\backslash theta$ with Fock-coupling regularisation architecture, scaled up to d=768, L=16 and trained on OpenWebText. This is not a final trained model — it is the diagnostic sweep used to pick the damping coefficient for a subsequent full 100,000-step training run at this scale. Each of the 8 checkpoints is included in full.

Alongside the perplexity sweep, every checkpoint is also scored with the damped-geodesic residual $\bar{R}(\gamma )\backslash bar\{R\}(\backslash gamma)$ — a closed-form diagnostic (no additional training, no autodiff through a learned metric) that measures how closely each model's own hidden-state trajectory follows a geodesic of the Riemannian (Jacobi) metric induced by its own learned potential. The headline result:

Both the perplexity minimum and the geodesic-residual minimum land at $\gamma =0.050\backslash gamma\; =\; 0.050$ — the smallest candidate tested, i.e. a boundary optimum rather than the interior minimum seen at d=384. This is also an exact, zero-parameter match to the two-regime closed-form damping predictor's high-\(d\) anchor prediction (\(\gamma^\ast_{\text{pred}} = 0.050\)) — the first time this predictor has been confirmed on a bounded anisotropic-Gaussian $V_{\theta }V\_\backslash theta$ at $d\ge 768d\; \backslash geq\; 768$, at a depth (\(L=16\)) the original MLP-\(V_\theta\) d=768 sweep never used. See Coincidence, Boundary, and an Exact Predictor Match.

Based on this sweep, a full 100,000-step run at $\gamma =0.05\backslash gamma\; =\; 0.05$ is expected to follow the same launch pattern as the d=384 run (currently in progress at $\gamma =0.10\backslash gamma=0.10$); as of this card's creation the d=768 full run has not yet been launched.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Repository
• Architecture
• The Gamma Sweep
• The Geodesic Residual Diagnostic
• Results: A Boundary Optimum at Gamma 0.05
• The Non-Monotonic Wiggle Past the Minimum
• Coincidence, Boundary, and an Exact Predictor Match
• Comparison to the d=384 Sibling
• Caveats: Short-Sweep Reliability
• How to Get Started
• Available Artifacts
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

When to Use This Repository

Use this repository if you want to:

• Reproduce or extend the gamma-selection methodology for the anisotropic-Gaussian + Fock-reg Fock-PARFLM line at d=768, L=16 on OpenWebText, including the geodesic-residual diagnostic.
• Study the non-monotonic PPL-vs-gamma wiggle that this width shares with the d=1024 sibling sweep but that no MLP-\(V_\theta\) sweep at comparable widths shows.
• Compare geodesic fidelity across damping regimes on a bounded, analytically-differentiable potential — the closed-form Jacobi-metric machinery here is structurally unavailable to attention-based or MLP-potential architectures.

Do not use this repository if you want a well-trained OpenWebText language model: every checkpoint here has seen only 3,000 steps (~24.6M tokens at effective batch 8 x block 512) and none is intended to produce fluent text. For a fully trained OpenWebText-scale Fock-PARFLM checkpoint, see semsimula-fock-parflm-depthcond-vtheta-openwebtext (27.23 PPL, isotropic Gaussian, d=384, 250K steps) — no full-length d=768 run in this line exists yet.

Architecture

Identical Fock-PARFLM v2.1 scaffold to the d=384 gamma-sweep sibling, scaled up to d=768 and, specific to this width, given explicit force bounding for numerical stability:

Input tokens x_1, ..., x_T
       |
   Untied token embedding E[x] + learned positional P[t]
       |
   For each of L=16 damped-Euler integration steps (shared force field):
       |
       +-- K=5 causal-EMA context channels:
       |     xi^(m)_t = causal_ema(h, alpha_m)      [horizons ~2 .. ~200 tokens]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian, force-bounded):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [5 contexts, 40 wells total]
       |     f_theta = clamp(-analytical_grad_h V_theta, max_norm=2/sqrt(768))
       |
       +-- Sparse pairwise V_phi (structural-competitive, 4 heads):
       |     top-k=16 past tokens per query (Gumbel routing)
       |     f_phi = -grad_h V_phi(h_t, h_s)                [autograd, sparse]
       |
       +-- Fock register pool (v2, 32 registers):
       |     M=32 virtual registers, Q/K/V creation gates, d_k=64
       |     LIFO stack discipline, per-register tau/keys, ortho init
       |     register repulsion (Gram penalty, lambda=0.05)
       |     reverse channel (per-layer, stabilised, pre-LN, soft-norm, warmup 4000)
       |     prefix-causal (leak-free by construction)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = clamp(f_theta + f_phi + f_fock, max_norm=2/sqrt(768))
       |
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       |
       +-- LayerNorm(h)
       |
   Logits = h @ W_out^T + b_out                              [UNTIED W_out]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

Parameter	Value
Hidden dim (d)	768
Layers (L)	16
Max sequence length	1024 (trained at block 512)
Vocab (GPT-2 BPE)	50257
V_theta kind	Depth-conditioned multi-context anisotropic Gaussian (bounded mixture)
V_theta contexts (n_ctx)	5 (one bank per xi channel)
Wells per context (K)	8
Total attractors	40
Anisotropic rank (r)	4 — low-rank factor B_k in R^(768x4) per well
Precision init / cap	a_k init at -log(768) (log-precision), capped at 2/768
Force bounding (d768-specific)	$V_{\theta }V\_\backslash theta$'s own gradient norm and the model-level total force are both clamped to $2\mathrm{/}\sqrt{768}\approx 0.07222/\backslash sqrt\{768\}\; \backslash approx\; 0.0722$ — not present in the d=384 sweep, added for stability at this width
Depth codes	per-layer additive shift, shape (L=16, n_ctx=5, d=768), init std 0.02
Xi channels (K_xi)	5 (alpha inits 0.50, 0.75, 0.95, 0.99, 0.995)
Fock-coupling regulariser	log-barrier on alpha_k, lambda=0.005, eps=1e-6 (see the TinyStories anchor for the full derivation)
V_phi kind	structural_competitive
V_phi heads / d_type / d_angle	4 / 32 / 16
Top-k (sparse routing)	16
Fock version	v2
Registers (M)	32
Register d_k	64
Stack discipline	LIFO
Reverse channel	Yes — per-layer, stabilised, pre-LN, soft-norm, 4,000-step warmup
Register repulsion	Gram penalty, lambda=0.05
Embeddings	Untied (separate W_out)
Mass model	logfreq (frozen OpenWebText surprisal lookup)
lambda_V (V_theta regularisation)	0.01
Prefix-causal registers	Yes — every checkpoint trained natively leak-free from step 0
Total parameters	225,354,011
V_theta parameters	141,834,280 (63% of total — the low-rank correction $B_k\in {\mathbb{R}}^{d\times r}B\_k\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; r\}$ scales with $dd$, so widening from d=384 to d=768 roughly quadruples $V_{\theta }V\_\backslash theta$'s share of the model)

The analytical form of $V_{\theta }V\_\backslash theta$ — the diagonal-plus-rank-4 precision ${\mathrm{\Sigma }}_k^{-1}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(a_k)+B_k{B}_k^{\mathrm{\top }}\backslash Sigma\_k^\{-1\}\; =\; \backslash mathrm\{diag\}(a\_k)\; +\; B\_k\; B\_k^\backslash top$, the depth-conditioning mechanism, and the closed-form bounded gradient — is unchanged from the TinyStories anchor and the d=384 sibling; only $dd$ changes, plus the explicit force-norm clamp described above.

The Gamma Sweep

Eight candidate damping coefficients $\gamma \in \{0.05,0.10,0.15,0.20,0.25,0.30,0.40,0.50\}\backslash gamma\; \backslash in\; \backslash \{0.05,\; 0.10,\; 0.15,\; 0.20,\; 0.25,\; 0.30,\; 0.40,\; 0.50\backslash \}$ were each trained from scratch for 3,000 steps (WSD schedule, peak LR 3e-4, effective batch 8, up to 1B-token training pool), then scored on a held-out 2M-token OpenWebText validation slice. This protocol mirrors the d=384 sweep and is designed to be cheap: 8 x 3,000 steps rather than 8 x 100,000. The batch/grad-accum split (2x4 here vs. 2x8 at d=384) and the tighter gradient-clip ceilings (0.5 global / 0.2 for V_phi, vs. 1.0 / 0.3 at d=384) are both d768-specific stability adjustments.

The Geodesic Residual Diagnostic

Fock-PARFLM's scalar potential $V_{\theta }V\_\backslash theta$ is a closed-form Gaussian mixture with an analytical gradient, which makes a diagnostic available here that is structurally unavailable to attention-based or MLP-potential architectures: at fixed energy $EE$, Hamiltonian trajectories are geodesics of the Jacobi metric $g_{ij}^J(x)=2(E-V(x)){\delta }_{ij}g^J\_\{ij\}(x)\; =\; 2(E\; -\; V(x))\backslash delta\_\{ij\}$, a conformally flat metric whose Christoffel symbols are closed-form functions of $\mathrm{\nabla }{V}_{\theta }\backslash nabla\; V\_\backslash theta$ — no learned metric, no autodiff through a metric, no boundary-value solve.

For a trajectory with position stream $x_{\mathrm{\ell }}x\_\backslash ell$, velocity stream $v_{\mathrm{\ell }}v\_\backslash ell$, and measured acceleration $a_{\mathrm{\ell }}a\_\backslash ell$ (the discrete second difference of $x_{\mathrm{\ell }}x\_\backslash ell$, consistent with the model's Velocity-Verlet-style integrator), the per-layer damped-geodesic residual is

$$R_{\mathrm{\ell }}=\frac{\parallel a_{\mathrm{\ell }}+\mathrm{\Gamma }(v_{\mathrm{\ell }},v_{\mathrm{\ell }})+\gamma v_{\mathrm{\ell }}\parallel }{\parallel a_{\mathrm{\ell }}\parallel +\epsilon },\mathrm{\Gamma }(v,v{)}^k={\mathrm{\Gamma }}_{ij}^k{v}^i{v}^j,R\_\backslash ell\; =\; \backslash frac\{\backslash big\backslash lVert\; a\_\backslash ell\; +\; \backslash Gamma(v\_\backslash ell,\; v\_\backslash ell)\; +\; \backslash gamma\; v\_\backslash ell\; \backslash big\backslash rVert\}\{\backslash lVert\; a\_\backslash ell\; \backslash rVert\; +\; \backslash varepsilon\},\; \backslash qquad\; \backslash Gamma(v,v)^k\; =\; \backslash Gamma^k\_\{ij\}\; v^i\; v^j,$$

where $\mathrm{\Gamma }\backslash Gamma$ is computed in closed form from $V_{\theta }V\_\backslash theta$'s analytical gradient. $R_{\mathrm{\ell }}\approx 0R\_\backslash ell\; \backslash approx\; 0$ means the trajectory is a damped geodesic of the metric induced by the model's own learned potential — this is not a pure-conservation claim (the explicit $\gamma v_{\mathrm{\ell }}\backslash gamma\; v\_\backslash ell$ damping term is included), only that the dynamics satisfy the damped geodesic equation with the architecture's own damping coefficient. Averaging over layers and 10 fixed validation batches (seed 42) gives $\bar{R}({\gamma }_{\text{train}})\backslash bar\{R\}(\backslash gamma\_\{\backslash text\{train\}\})$, evaluated at ${\gamma }_{\text{eval}}={\gamma }_{\text{train}}\backslash gamma\_\{\backslash text\{eval\}\}\; =\; \backslash gamma\_\{\backslash text\{train\}\}$ for each retained checkpoint — the diagonal overlay against $\mathrm{P}\mathrm{P}\mathrm{L}({\gamma }_{\text{train}})\backslash mathrm\{PPL\}(\backslash gamma\_\{\backslash text\{train\}\})$.

A second, closed-form quantity — the recovered intrinsic damping ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ — is the least-squares row minimiser "the damping value that best explains this specific trajectory," independent of what $\gamma \backslash gamma$ the model was actually trained with:

$${\gamma }_{\text{geo}}=-\frac{⟨a_{\mathrm{\ell }}+\mathrm{\Gamma }(v_{\mathrm{\ell }},v_{\mathrm{\ell }}),v_{\mathrm{\ell }}⟩}{\parallel v_{\mathrm{\ell }}{\parallel }^2}\mathrm{.}\backslash gamma\_\{\backslash text\{geo\}\}\; =\; -\backslash frac\{\backslash big\backslash langle\; a\_\backslash ell\; +\; \backslash Gamma(v\_\backslash ell,\; v\_\backslash ell),\backslash \; v\_\backslash ell\; \backslash big\backslash rangle\}\{\backslash lVert\; v\_\backslash ell\; \backslash rVert^\{2\}\}.$$

Full derivation, practical mitigations (turning-point exclusion, reference-energy convention, integrator staggering), and validation controls (vanilla-baseline, shuffled-\(\Gamma\), and random-direction nulls) are in the companion note Geodesic_Preservation_Experiment.md.

Results: A Boundary Optimum at Gamma 0.05

$\gamma \backslash gamma$	PPL	$\bar{R}\backslash bar\{R\}$	${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$	Excluded frac
0.050	326.97 ← PPL min	1.2813 ← $\bar{R}\backslash bar\{R\}$ min	0.9829	0%
0.100	350.70	1.9648	0.9828	0%
0.150	340.25	1.5159	0.9827	0%
0.200	368.62	1.7544	0.9795	0%
0.250	362.39	2.9092	0.9840	0%
0.300	362.44	1.4901	0.9735	0%
0.400	393.47	3.5790	0.9848	0%
0.500	364.94	2.6750	0.9781	0%

The minima coincide, at the boundary of the swept range. Both PPL and $\bar{R}\backslash bar\{R\}$ bottom out at $\gamma =0.050\backslash gamma\; =\; 0.050$, the smallest gamma tested — unlike the d=384 sweep, where the joint minimum sat at an interior point (\(0.100\)) with worse candidates on both sides. ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ also shows the tightest cross-gamma clustering of any sweep in this family to date: mean $0.98100.9810$, std $0.00350.0035$ — essentially flat across a full order-of-magnitude range of nominal training $\gamma \backslash gamma$ (0.05 to 0.50), the same "intrinsic preferred geometry" signature documented at every other scale, but here with less than half the spread seen at d=384.

The per-layer residual heatmap below shows where in the network the departures from geodesic behaviour concentrate at each damping level — notice the bright band across nearly every layer at $\gamma =0.40\backslash gamma=0.40$, the sweep's worst PPL candidate, and the uniformly dark (near-geodesic) row at $\gamma =0.05\backslash gamma=0.05$:

The recovered intrinsic damping is visually almost a flat line across the entire swept range:

The Non-Monotonic Wiggle Past the Minimum

Unlike the original MLP-\(V_\theta\) d=768 sweep (a clean, monotonic PPL increase from $\gamma =0.05\backslash gamma=0.05$ to $0.500.50$; see Fock-PARFLM_Scale-Up_Gamma_Sweep_Results_and_Damping_Regime_Analysis.md), this anisotropic-Gaussian sweep is non-monotonic past its minimum: PPL rises from 326.97 (\(\gamma=0.05\)) to a local peak at $\gamma =0.20\backslash gamma=0.20$ (368.62), dips back down at $\gamma =0.25\backslash gamma=0.25$-\(0.30\) (362.39, 362.44), rises again to the sweep's worst point at $\gamma =0.40\backslash gamma=0.40$ (393.47), then partially recovers at $\gamma =0.50\backslash gamma=0.50$ (364.94).

The overall ranking is not in doubt — $\gamma =0.05\backslash gamma=0.05$ leads its nearest competitor (\(\gamma=0.15\), 340.25 PPL) by 13.28 PPL, about 4% — but the wiggle itself is a genuine feature of the aniso-Gaussian + Fock-reg configuration that no MLP-\(V_\theta\) sweep at comparable widths shows. A companion sweep at d=1024 (not yet uploaded to this collection) shows the same wiggle shape, with a local peak in the same \gamma \approx 0.20\)-\\(0.40 range before a partial recovery at 0.40-0.50 — a weak but real argument against pure single-run noise, since independent per-candidate noise would not obviously line up at matching $\gamma \backslash gamma$ values across two different widths. Two candidate explanations remain open (not mutually exclusive): (1) single-seed optimisation noise riding on a genuinely flat or slowly-varying underlying PPL surface, or (2) a structural interaction between the anisotropic precision matrices (or the reverse channel) and mid-range explicit friction. Disentangling them would require a 2-3 seed rerun of one or two candidates (e.g. $\gamma =0.30\backslash gamma=0.30$) — not yet done. See Determining_optimal_gamma_for_Fock-PARFLM.md §14.5 for the full cross-scale discussion.

Coincidence, Boundary, and an Exact Predictor Match

The two-regime closed-form damping predictor, calibrated on the original (MLP) $V_{\theta }V\_\backslash theta$ family, gives at $d=768d=768$, $L=16L=16$, $\bar{m}=1.4\backslash bar\{m\}=1.4$ (using the high-\(d\) regime constant ${\rho }_{\text{hi}}=0.565\backslash rho\_\{\backslash text\{hi\}\}\; =\; 0.565$):

$${\gamma }_{\text{pred}}^{\ast }=\frac{1.4}{16}\ln (1\mathrm{/}0.565)=0.0875\times 0.571=0.050\backslash gamma^\{\backslash ast\}\_\{\backslash text\{pred\}\}\; =\; \backslash frac\{1.4\}\{16\}\backslash ln(1/0.565)\; =\; 0.0875\; \backslash times\; 0.571\; =\; 0.050$$

Empirical ${\gamma }^{\ast }=0.050\backslash gamma^\{\backslash ast\}\; =\; 0.050$. Exact match, zero error. This resolves an open question left by the d=384 sweep — whether the aniso-Gaussian family's crossover, once past $d=384d=384$, matches the high-\(d\) anchor exactly (as the MLP-\(V_\theta\) family does) or retains a residual offset. It matches exactly, and at a depth (\(L=16\)) the original MLP $d=768d=768$ sweep never tested (that sweep used $L=12L=12$, where the depth-only formula overshoots by 33% — a known, pre-existing discrepancy unrelated to this architecture). Combined with the same exact match found in the d=1024 companion sweep, this is the strongest evidence to date that ${\rho }_{\text{hi}}=0.565\backslash rho\_\{\backslash text\{hi\}\}=0.565$ is a genuine architecture-family invariant at $d\gtrsim 768d\; \backslash gtrsim\; 768$, not an artefact of the MLP $V_{\theta }V\_\backslash theta$ or of the specific depths swept when the constant was first calibrated.

Sweep	$dd$	$LL$	$V_{\theta }V\_\backslash theta$	${\gamma }_{\text{pred}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash text\{pred\}\}$	${\gamma }_{\text{empirical}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash text\{empirical\}\}$	Match
Original MLP d=768 sweep	768	12	MLP	0.067	0.05	+33% (known $L=12L=12$ depth-formula overshoot)
This sweep	768	16	aniso-Gaussian + fock-reg	0.050	0.050	Exact
d=1024 companion (not yet uploaded)	1024	16	aniso-Gaussian + fock-reg	0.050	0.050	Exact

Comparison to the d=384 Sibling

Sweep	${\gamma }_{\text{PPL}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash text\{PPL\}\}$	${\gamma }_{\bar{R}}^{\ast }\backslash gamma^\backslash ast\_\{\backslash bar\{R\}\}$	Coincidence?	Optimum shape	Margin vs. runner-up
d=384, L=16	0.100	0.100	Yes — gap 0	Interior minimum	large
d=768, L=16 (this sweep)	0.050	0.050	Yes — gap 0	Boundary minimum	~4%, modest

Both widths show coinciding PPL and geodesic-residual minima — extending the pattern first seen at d=384 to a second, larger width — but the shape of the optimum differs: d=384's winner sits strictly between two worse neighbours (a genuine interior bowl, only slightly undermined by the $\gamma =0.20\backslash gamma=0.20$ divergence outlier there), while d=768's winner is the smallest gamma tested, with PPL and $\bar{R}\backslash bar\{R\}$ both rising (non-monotonically) as gamma increases from there. The ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ clustering is also tighter at d=768 (std 0.0035 vs. a wider spread at d=384, driven partly by that sweep's $\gamma =0.20\backslash gamma=0.20$ outlier) — consistent with a model that has, if anything, an even more uniform intrinsic damping preference at this width.

Caveats: Short-Sweep Reliability

This is a 3,000-step, single-seed measurement per gamma, and the family has one documented case where a short-sweep ranking reversed at full training length: the d=256 TinyStories aniso-Gaussian sweep favoured $\gamma =0.150\backslash gamma=0.150$ at 3K steps, but the full 20K-step run favoured $\gamma =0.300\backslash gamma=0.300$ instead. Two considerations specific to this sweep are worth weighing before trusting $\gamma =0.05\backslash gamma=0.05$ at a full 100K-step horizon:

• The margin between $\gamma =0.05\backslash gamma=0.05$ and its nearest competitor (\(\gamma=0.15\)) is real but modest — about 4%, right at the edge of the family's informal "flat bowl, ranking unreliable" threshold (5%) from the earlier d=256 reversal analysis. The d=1024 companion sweep's margin (10-11%) is comfortably wider.
• The non-monotonic wiggle documented above shows this configuration is not perfectly smooth in gamma, so the training log for a full run should still be watched for instability signatures analogous to the d=384 sweep's $\gamma =0.20\backslash gamma=0.20$ divergence-and-partial-recovery excursion, as an early-warning check.

Two considerations nonetheless favour trusting this sweep's recommendation more than a typical modest-margin case: (a) unlike the flat interior bowls at $d\le 384d\; \backslash leq\; 384$ where the documented reversal actually occurred, this is a boundary optimum with ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ essentially flat across the whole range — there is no competing interior basin the sweep could be misranking within, only a preference for less explicit friction that a longer horizon is unlikely to overturn; and (b) the predictor's independent, zero-parameter prediction lands on the exact same value (above). Recommendation: $\gamma =0.05\backslash gamma=0.05$ for the d=768, L=16 aniso-Gaussian full run.

How to Get Started

import math, torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing
from huggingface_hub import hf_hub_download

REPO = "dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep"
GAMMA = "0.050"  # the recommended candidate; also available: 0.100, 0.150, 0.200, 0.250, 0.300, 0.400, 0.500

logfreq_path = hf_hub_download(repo_id=REPO, filename="results/logfreq_surprisal_openwebtext.npy")
FORCE_MAX = 2.0 / math.sqrt(768)

config = FockMultiXiPARFConfig(
    vocab_size=50257, d=768, max_len=1024, L=16,
    v_hidden=1024, v_depth=3, dt=1.0,
    mass_mode="logfreq", logfreq_path=logfreq_path, logfreq_init_alpha=0.1,
    init_gamma=1.0, fixed_gamma=float(GAMMA),
    causal_force=True, ln_after_step=True,
    xi_channels=5, xi_alpha_inits=[0.50, 0.75, 0.95, 0.99, 0.995],
    xi_learnable=True, xi_alpha_init_mode="explicit",
    fock_version="v2", n_registers=32,
    reverse_channel=True, reverse_channel_stable=True, reverse_channel_pre_ln=True,
    reverse_channel_soft_norm=True, reverse_channel_warmup_steps=4000, reverse_channel_per_layer=True,
    register_repulsion=True, register_repulsion_coeff=0.05,
    prefix_causal_registers=True,
    v_phi_kind="structural_competitive", v_phi_n_heads=4, v_phi_d_type=32, v_phi_d_angle=16,
    v_phi_eps=0.1, v_phi_phi_hidden=128, v_phi_theta_hidden=128, v_phi_mlp_hidden=128,
    top_k=16,
    use_output_bias=True, tie_embeddings=False,
    score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3, gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True, per_layer_v_phi_scale=True,
    register_salience_decay=0.5, register_salience_threshold=0.005,
    creation_gate_hidden=64, stack_discipline=True,
    d_k=64, tau_create_init=8.0,
    per_register_tau=True, per_register_keys=True,
    ortho_register_init=True,
    force_clamp_max=FORCE_MAX,
)
model = FockMultiXiPARFLM(config)

model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=768, K=8, n_ctx=5, n_layers=16, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(768),
    precision_max=2.0 / 768,
    force_norm_max=FORCE_MAX,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)

ckpt_path = hf_hub_download(repo_id=REPO, filename=f"checkpoints/gamma_{GAMMA}/ckpt_best.pt")
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")            # 225,354,011
print(f"gamma={state['gamma']}  step={state['step']:,}  val_ppl={state['val_ppl']:.2f}")

Available Artifacts

Path	Description
checkpoints/gamma_0.050/ckpt_best.pt ... checkpoints/gamma_0.500/ckpt_best.pt	All 8 gamma-sweep checkpoints (step 3,000 each), each a dict with model_state_dict, optimizer_state_dict, step, val_loss, val_ppl, gamma, v_theta_variant, aniso_rank, lambda_fock_reg
checkpoints/gamma_*/training_log.jsonl	Per-500-step training metrics and the step-2,000 causal_probe event, one file per gamma
sweep_summary.json	Sweep-level summary (best PPL per gamma, wall-clock)
results/geodesic_results.json	Full per-gamma geodesic residual data: $\bar{R}\backslash bar\{R\}$, ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$, excluded fraction, and all 15 per-layer residuals
results/geodesic_overlay_aniso_gaussian_d768.png	PPL vs. geodesic residual dual-axis overlay
results/geodesic_per_layer_aniso_gaussian_d768.png	Per-layer, per-gamma geodesic residual heatmap
results/gamma_geo_recovery_aniso_gaussian_d768.png	Recovered intrinsic damping ${\gamma }_{\text{geo}}\backslash gamma\_\{\backslash text\{geo\}\}$ vs. training gamma
results/logfreq_surprisal_openwebtext.npy	Frozen per-token log-frequency (surprisal) lookup used by the logfreq mass model
model_aniso_gaussian_vtheta.py	Anisotropic Gaussian V_theta classes + install_aniso_depth_routing (identical file to the TinyStories anchor's and the d=384 sibling's)
config.json	Full sweep configuration, per-gamma results table, and predictor comparison

Training Details

Training Data

OpenWebText, tokenized with GPT-2 BPE (vocab 50257). Each candidate trains on up to 1B tokens (early stopped at 3,000 steps, effective batch 8, block 512 -> ~12.3M tokens actually consumed) and is evaluated on a held-out 2M-token validation slice with no train/val overlap.

Training Procedure (per gamma candidate)

Hyperparameter	Value
Optimizer	AdamW (betas 0.9/0.95, weight decay 0.01)
LR schedule	WSD (warmup 0-150, stable 150-1950, decay 1950-3000)
Peak learning rate	3e-4
Batch size	2 x grad-accum 4 (effective 8)
Block size	512
Steps	3,000
Gradient clipping	per-group (global 0.5; V_phi 0.2 — tighter than d=384's 1.0/0.3, a d768-specific stability adjustment)
Force clamp (model-level and V_theta-level)	$2\mathrm{/}\sqrt{768}\approx 0.07222/\backslash sqrt\{768\}\; \backslash approx\; 0.0722$
lambda_V (V_theta regularisation)	0.01
lambda_fock (coupling regulariser)	0.005
Seed	0
Hardware	1x NVIDIA H100/A100 (Google Colab)

Causal-Leak Verification

All 8 checkpoints were trained natively with prefix_causal_registers=True from step 0. The bit-exact future-perturbation causal probe passed with max_delta=0.0 at step 2,000 for every one of the 8 gamma candidates.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_gamma_sweep_geodesic_aniso_gaussian_fockreg_d768.ipynb (companion repo) — self-contained Colab notebook that runs the 8-candidate sweep, the geodesic residual analysis, and produces the overlay/heatmap/recovery figures in one pass, requiring no additional training beyond the sweep itself.

Evaluation Results

OpenWebText Validation Perplexity (3,000-step sweep candidates)

$\gamma \backslash gamma$	PPL	Rank
0.050	326.97	1st
0.150	340.25	2nd
0.300	362.44	3rd
0.250	362.39	4th
0.500	364.94	5th
0.100	350.70	6th
0.200	368.62	7th
0.400	393.47	8th

These PPL values are from 3,000-step short-sweep candidates and are not comparable to the fully trained OpenWebText checkpoints elsewhere in this family (e.g. 27.23 PPL after 250K steps for the isotropic d=384 flagship). They exist solely to rank candidate damping coefficients. Note the ranking is not simply increasing with $\gamma \backslash gamma$; see The Non-Monotonic Wiggle Past the Minimum.

See Results: A Boundary Optimum at Gamma 0.05 for the combined PPL / geodesic-residual table.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Model	Design	Corpus	PPL	HuggingFace
Multi-Xi SPLM (MLP)	Pure scalar potential	TinyStories	11.51	semsimula-splm-multixi
Multi-Xi PARFLM (MLP)	Scalar + pairwise forces	TinyStories	12.06	semsimula-parflm-multixi
Fock-PARFLM v2.1 (MLP)	PARFLM + Fock registers	TinyStories	9.70	semsimula-fock-parflm
Fock-PARFLM v2.1 (SQ3)	Structured + pairwise + Fock	TinyStories	10.90	semsimula-fock-parflm-structured-vtheta
Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian)	Bounded multi-context + pairwise + Fock	TinyStories	16.33	semsimula-fock-parflm-depthcond-vtheta
Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian + fock-reg)	Bounded, ellipsoidal multi-context + pairwise + Fock	TinyStories	9.04	semsimula-fock-parflm-anisogaussian-vtheta
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, first-order/Fock-G1)	Same as above, gradient-flow integrator	TinyStories	8.95	semsimula-fock-parflm-anisogaussian-vtheta-fock-g1
Fock-PARFLM v2.1 (depth-cond. Gaussian, OpenWebText scale)	Bounded multi-context + pairwise + Fock, d=384 L=16	OpenWebText	27.23	semsimula-fock-parflm-depthcond-vtheta-openwebtext
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=384)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	278.27 (best of 8, 3K-step sweep, not a final model)	semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep
Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=768)	Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis	OpenWebText	326.97 (best of 8, 3K-step sweep, not a final model)	this repository
Fock-Attention	Fock + attention	TinyStories	9.42	semsimula-fock-attention
Hybrid SPLM+Attn	Attention + SPLM refinement	TinyStories	8.50	semsimula-hybrid-splm

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Not a final trained model. Every checkpoint in this repository has been trained for only 3,000 steps — a gamma-selection diagnostic, not a language model intended for generation or downstream use. Do not compare its PPL to fully trained checkpoints elsewhere in this family.
• Single-seed sweep. Each gamma candidate is one run; no seed-variance estimate is available. The margin between $\gamma =0.05\backslash gamma=0.05$ and its nearest competitor (~4%) is real but modest; see Caveats: Short-Sweep Reliability.
• Short-horizon ranking can reverse. The family has one documented case (d=256 TinyStories) where the short-sweep-optimal gamma differed from the full-run-optimal gamma. The $\gamma =0.05\backslash gamma=0.05$ recommendation here carries some of the same risk, mitigated by the boundary-optimum shape and the exact predictor match; see the caveats section.
• Non-monotonic PPL-vs-gamma wiggle, shared with the d=1024 companion sweep, is not yet explained (single-seed noise vs. a structural property of this $V_{\theta }V\_\backslash theta$ configuration); see above.
• OpenWebText only, English only. No instruction tuning, no RLHF/DPO, no safety filtering.
• Geodesic residual is a diagnostic, not a training objective. The model was trained to minimise cross-entropy; the near-geodesic behaviour at $\gamma =0.05\backslash gamma=0.05$ is an emergent structural property, not something the loss function directly optimises for.
• Very large $V_{\theta }V\_\backslash theta$ hypernetwork. At 141.8M parameters, $V_{\theta }V\_\backslash theta$ alone is 63% of the 225M-parameter total — a substantially larger share than the d=384 sibling's 46%, driven by the low-rank correction $B_k\in {\mathbb{R}}^{d\times r}B\_k\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; r\}$ scaling with $dd$.
• No causal-leak issue. All 8 checkpoints were trained natively with prefix_causal_registers=True; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Companion code repository:
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: 1x NVIDIA H100/A100 80GB (Google Colab)
• Training: 8 x 3,000 steps = 24,000 total training steps across the sweep, plus inference-only geodesic residual analysis (10 validation batches x 8 checkpoints, no additional training)
• Carbon footprint: small; a single-GPU research sweep, estimated on the order of a few kg CO2