---
pipeline_tag: text-generation
library_name: pytorch
language:
- en
license: cc-by-4.0
tags:
- research
- semsimula
- conservative-language-model
- scalar-potential
- lagrangian-mechanics
- energy-based-model
- physics-informed
- fock-space
- parflm
- parf
- multi-channel-xi
- depth-conditioned-vtheta
- multi-context-vtheta
- anisotropic-gaussian-potential
- low-rank-precision
- gaussian-mixture-potential
- bounded-potential
- analytical-gradient
- interpretable
- virtual-registers
- reverse-channel
- controlled-conservativity
- sparse-routing
- openwebtext
- gamma-sweep
- damping-coefficient
- non-transformer
- attention-free
- constant-memory-inference
- riemannian-geometry
- riemannian-geodesics
- geodesic-residual
- jacobi-metric
datasets:
- Skylion007/openwebtext
metrics:
- perplexity
model-index:
- name: semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep
results:
- task:
type: text-generation
dataset:
type: Skylion007/openwebtext
name: OpenWebText
split: validation
metrics:
- type: perplexity
value: 244.23
name: Validation Perplexity (gamma=0.05, best of 8 gamma-sweep candidates, 3,000-step short sweep — not a fully trained model)
verified: false
---
# Fock-PARFLM v2.1 Anisotropic Gaussian V_theta + Fock Regularisation — Gamma Sweep with Geodesic Residual Analysis (OpenWebText, d=1024)
This repository holds **eight short (3,000-step) training runs**, one per candidate damping coefficient \\(\gamma \in \{0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.40, 0.50\}\\), of the [depth-conditioned anisotropic Gaussian \\(V_\theta\\) with Fock-coupling regularisation](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta) architecture, scaled up to **d=1024, L=16** and trained on **OpenWebText**. This is **not a final trained model** — it is the diagnostic sweep used to pick the damping coefficient for a subsequent full 100,000-step training run at this scale. Each of the 8 checkpoints is included in full, and (unlike the [isotropic-Gaussian d=1024 sweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-depthcond-vtheta-openwebtext) elsewhere in this family, which only completed 4/8 candidates) all 8 candidates here completed successfully.
Alongside the perplexity sweep, every checkpoint is also scored with the **damped-geodesic residual** \\(\bar{R}(\gamma)\\) — a closed-form diagnostic (no additional training, no autodiff through a learned metric) that measures how closely each model's own hidden-state trajectory follows a geodesic of the Riemannian (Jacobi) metric induced by its own learned potential. The headline result:
> **Both the perplexity minimum and the geodesic-residual minimum land at \\(\gamma = 0.050\\)** — the smallest candidate tested, the same **boundary optimum** shape found at [d=768](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep), but here with the **widest margin of any boundary-optimum sweep in the family** (~10.5% over the runner-up) and, unlike d=768, a geodesic-residual curve that rises essentially **monotonically** past the minimum. This is also an **exact, zero-parameter match** to the two-regime closed-form damping predictor's high-\\(d\\) anchor prediction — the second consecutive exact confirmation at \\(d \geq 768\\) after d=768, and the third overall counting the original MLP architecture's own d=1024 result. See [Coincidence, Boundary, and a Second Exact Predictor Match](#coincidence-boundary-and-a-second-exact-predictor-match).
Based on this sweep, a full 100,000-step run at \\(\gamma = 0.05\\) is expected to follow the same launch pattern as the [d=384](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep) and (pending) d=768 runs; as of this card's creation the d=1024 full run has **not yet been launched**.
This model is from the [Semantic Simulation](https://doi.org/10.5281/zenodo.19712427) framework.
## Table of Contents
- [When to Use This Repository](#when-to-use-this-repository)
- [Architecture](#architecture)
- [The Gamma Sweep](#the-gamma-sweep)
- [The Geodesic Residual Diagnostic](#the-geodesic-residual-diagnostic)
- [Results: The Widest Boundary-Optimum Margin Yet](#results-the-widest-boundary-optimum-margin-yet)
- [The PPL Wiggle Without the Geodesic Wiggle](#the-ppl-wiggle-without-the-geodesic-wiggle)
- [Coincidence, Boundary, and a Second Exact Predictor Match](#coincidence-boundary-and-a-second-exact-predictor-match)
- [Comparison Across All Three Widths](#comparison-across-all-three-widths)
- [Caveats: Short-Sweep Reliability and Two Incomplete Training Logs](#caveats-short-sweep-reliability-and-two-incomplete-training-logs)
- [How to Get Started](#how-to-get-started)
- [Available Artifacts](#available-artifacts)
- [Training Details](#training-details)
- [Evaluation Results](#evaluation-results)
- [SPLM Family Overview](#splm-family-overview)
- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
- [Citation](#citation)
- [Environmental Impact](#environmental-impact)
## When to Use This Repository
Use this repository if you want to:
- **Reproduce or extend the gamma-selection methodology** for the anisotropic-Gaussian + Fock-reg Fock-PARFLM line at d=1024, L=16 on OpenWebText, including the geodesic-residual diagnostic.
- **Study the PPL-only non-monotonic wiggle**: at this width the PPL-vs-gamma curve wiggles (as at [d=768](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep#the-non-monotonic-wiggle-past-the-minimum)) but the geodesic-residual curve does **not** — see [below](#the-ppl-wiggle-without-the-geodesic-wiggle).
- **Compare geodesic fidelity across damping regimes** on a bounded, analytically-differentiable potential — the closed-form Jacobi-metric machinery here is structurally unavailable to attention-based or MLP-potential architectures.
- **Study the widest and cleanest boundary optimum in the family**, useful as the "textbook case" for what a reliable short-sweep result looks like, in contrast to the flat-bowl / reversal-prone cases at \\(d \leq 384\\).
Do **not** use this repository if you want a well-trained OpenWebText language model: every checkpoint here has seen only 3,000 steps (~50M tokens at effective batch 16 x block 512) and none is intended to produce fluent text. For a fully trained OpenWebText-scale Fock-PARFLM checkpoint, see [semsimula-fock-parflm-depthcond-vtheta-openwebtext](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-depthcond-vtheta-openwebtext) (27.23 PPL, isotropic Gaussian, d=384, 250K steps) — no full-length d=1024 run in this line exists yet.
## Architecture
Identical Fock-PARFLM v2.1 scaffold to the [d=384](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep) and [d=768](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep) gamma-sweep siblings, scaled up to d=1024, with the same d768-style force-bounding for numerical stability plus a further-lowered peak learning rate:
```
Input tokens x_1, ..., x_T
|
Untied token embedding E[x] + learned positional P[t]
|
For each of L=16 damped-Euler integration steps (shared force field):
|
+-- K=5 causal-EMA context channels:
| xi^(m)_t = causal_ema(h, alpha_m) [horizons ~2 .. ~200 tokens]
|
+-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian, force-bounded):
| xi_g^(m) = xi^(m) + e_g^(m) [per-layer depth code]
| diff_k^m = h - mu_k^m(xi_g^(m))
| V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
| V_theta = sum_m V_m(xi_g^(m), h) [5 contexts, 40 wells total]
| f_theta = clamp(-analytical_grad_h V_theta, max_norm=2/sqrt(1024))
|
+-- Sparse pairwise V_phi (structural-competitive, 4 heads):
| top-k=16 past tokens per query (Gumbel routing)
| f_phi = -grad_h V_phi(h_t, h_s) [autograd, sparse]
|
+-- Fock register pool (v2, 32 registers):
| M=32 virtual registers, Q/K/V creation gates, d_k=64
| LIFO stack discipline, per-register tau/keys, ortho init
| register repulsion (Gram penalty, lambda=0.05)
| reverse channel (per-layer, stabilised, pre-LN, soft-norm, warmup 4000)
| prefix-causal (leak-free by construction)
| f_fock = creation + destruction + exchange forces
|
+-- Total force: f = clamp(f_theta + f_phi + f_fock, max_norm=2/sqrt(1024))
|
+-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
|
+-- LayerNorm(h)
|
Logits = h @ W_out^T + b_out [UNTIED W_out]
Auxiliary training-only loss term (not part of the forward pass above):
L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps) [log-barrier on xi coupling]
```
| Parameter | Value |
|---|---|
| Hidden dim (d) | 1024 |
| Layers (L) | 16 |
| Max sequence length | 1024 (trained at block 512) |
| Vocab (GPT-2 BPE) | 50257 |
| V_theta kind | Depth-conditioned multi-context **anisotropic** Gaussian (bounded mixture) |
| V_theta contexts (n_ctx) | 5 (one bank per xi channel) |
| Wells per context (K) | 8 |
| Total attractors | 40 |
| Anisotropic rank (r) | 4 — low-rank factor `B_k` in `R^(1024x4)` per well |
| Precision init / cap | `a_k` init at `-log(1024)` (log-precision), capped at `2/1024` |
| Force bounding | \\(V_\theta\\)'s own gradient norm and the model-level total force are both clamped to \\(2/\sqrt{1024} = 0.0625\\) (same style introduced at d=768; not present in the d=384 sweep) |
| Depth codes | per-layer additive shift, shape (L=16, n_ctx=5, d=1024), init std 0.02 |
| Xi channels (K_xi) | 5 (`alpha` inits 0.50, 0.75, 0.95, 0.99, 0.995) |
| Fock-coupling regulariser | log-barrier on alpha_k, lambda=0.005, eps=1e-6 (see the [TinyStories anchor](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta#the-fock-coupling-regularisation-term) for the full derivation) |
| V_phi kind | structural_competitive |
| V_phi heads / d_type / d_angle | 4 / 32 / 16 |
| Top-k (sparse routing) | 16 |
| Fock version | v2 |
| Registers (M) | 32 |
| Register d_k | 64 |
| Stack discipline | LIFO |
| Reverse channel | Yes — per-layer, stabilised, pre-LN, soft-norm, 4,000-step warmup |
| Register repulsion | Gram penalty, lambda=0.05 |
| Embeddings | Untied (separate `W_out`) |
| Mass model | logfreq (frozen OpenWebText surprisal lookup) |
| lambda_V (V_theta regularisation) | 0.01 |
| Prefix-causal registers | **Yes** — every checkpoint trained natively leak-free from step 0 |
| Total parameters | **363,884,315** |
| V_theta parameters | **252,026,920** (69% of total — the largest \\(V_\theta\\) share yet in this family; the low-rank correction \\(B_k \in \mathbb{R}^{d \times r}\\) scales with \\(d\\), so widening from d=768 to d=1024 pushes \\(V_\theta\\)'s share from 63% to 69%) |
The analytical form of \\(V_\theta\\) — the diagonal-plus-rank-4 precision \\(\Sigma_k^{-1} = \mathrm{diag}(a_k) + B_k B_k^\top\\), the depth-conditioning mechanism, and the closed-form bounded gradient — is unchanged from the [TinyStories anchor](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta#the-analytical-form-of-the-potential) and the [d=384](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep#architecture)/[d=768](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep#architecture) siblings; only \\(d\\) changes, plus the force-norm clamp inherited from d=768.
## The Gamma Sweep
Eight candidate damping coefficients \\(\gamma \in \{0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.40, 0.50\}\\) were each trained from scratch for 3,000 steps (WSD schedule, peak LR **1.5e-4** — half of the d384/d768 sweeps' 3e-4, a d1024-specific stability adjustment — effective batch 16, up to 1B-token training pool), then scored on a held-out 2M-token OpenWebText validation slice. This protocol mirrors the [d=384](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep#the-gamma-sweep) and [d=768](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep#the-gamma-sweep) sweeps and is designed to be cheap: 8 x 3,000 steps rather than 8 x 100,000. Unlike d=768's batch=2 x accum=4, this sweep reverts to batch=1 x accum=16 (effective 16, matching d=384) with the wider global/V_phi gradient-clip ceilings (1.0/0.3) also matching d=384 rather than d=768's tighter 0.5/0.2 — the halved peak LR was evidently sufficient stabilisation at this width on its own.
## The Geodesic Residual Diagnostic
Fock-PARFLM's scalar potential \\(V_\theta\\) is a closed-form Gaussian mixture with an analytical gradient, which makes a diagnostic available here that is structurally unavailable to attention-based or MLP-potential architectures: at fixed energy \\(E\\), Hamiltonian trajectories are geodesics of the **Jacobi metric** \\(g^J_{ij}(x) = 2(E - V(x))\delta_{ij}\\), a conformally flat metric whose Christoffel symbols are closed-form functions of \\(\nabla V_\theta\\) — no learned metric, no autodiff through a metric, no boundary-value solve.
For a trajectory with position stream \\(x_\ell\\), velocity stream \\(v_\ell\\), and measured acceleration \\(a_\ell\\) (the discrete second difference of \\(x_\ell\\), consistent with the model's Velocity-Verlet-style integrator), the **per-layer damped-geodesic residual** is
$$
R_\ell = \frac{\big\lVert a_\ell + \Gamma(v_\ell, v_\ell) + \gamma v_\ell \big\rVert}{\lVert a_\ell \rVert + \varepsilon}, \qquad \Gamma(v,v)^k = \Gamma^k_{ij} v^i v^j,
$$
where \\(\Gamma\\) is computed in closed form from \\(V_\theta\\)'s analytical gradient. \\(R_\ell \approx 0\\) means the trajectory is a damped geodesic of the metric induced by the model's own learned potential — this is not a pure-conservation claim (the explicit \\(\gamma v_\ell\\) damping term is included), only that the dynamics satisfy the damped geodesic equation with the architecture's own damping coefficient. Averaging over layers and 10 fixed validation batches (seed 42) gives \\(\bar{R}(\gamma_{\text{train}})\\), evaluated at \\(\gamma_{\text{eval}} = \gamma_{\text{train}}\\) for each retained checkpoint — the diagonal overlay against \\(\mathrm{PPL}(\gamma_{\text{train}})\\).
A second, closed-form quantity — the **recovered intrinsic damping** \\(\gamma_{\text{geo}}\\) — is the least-squares row minimiser "the damping value that best explains this specific trajectory," independent of what \\(\gamma\\) the model was actually trained with:
$$
\gamma_{\text{geo}} = -\frac{\big\langle a_\ell + \Gamma(v_\ell, v_\ell),\ v_\ell \big\rangle}{\lVert v_\ell \rVert^{2}}.
$$
Full derivation, practical mitigations (turning-point exclusion, reference-energy convention, integrator staggering), and validation controls (vanilla-baseline, shuffled-\\(\Gamma\\), and random-direction nulls) are in the companion note [`Geodesic_Preservation_Experiment.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Geodesic_Preservation_Experiment.md).
## Results: The Widest Boundary-Optimum Margin Yet
| \\(\gamma\\) | PPL | \\(\bar{R}\\) | \\(\gamma_{\text{geo}}\\) | Excluded frac |
|---:|---:|---:|---:|---:|
| **0.050** | **244.23** ← PPL min | **1.2066** ← \\(\bar{R}\\) min | 0.9580 | 0% |
| 0.100 | 270.80 | 1.3139 | 0.9690 | 0% |
| 0.150 | 264.77 | 1.4842 | 0.9628 | 0% |
| 0.200 | 288.50 | 1.5201 | 0.9643 | 0% |
| 0.250 | 284.46 | 1.5358 | 0.9632 | 0% |
| 0.300 | 295.05 | 1.8097 | 0.9622 | 0% |
| 0.400 | 298.05 | 2.0380 | 0.9613 | 0% |
| 0.500 | 269.77 | 2.2954 | 0.9606 | 0% |
**The minima coincide, at the boundary of the swept range, with the widest margin yet.** Both PPL and \\(\bar{R}\\) bottom out at \\(\gamma = 0.050\\); \\(\gamma=0.05\\) beats its nearest PPL competitors by \\(\approx10\text{-}11\%\\) (25.5-26.6 PPL against \\(\gamma=0.50\\) and \\(\gamma=0.10\\) respectively) — comfortably wider than the [d=768 sweep's \\(\approx4\%\\) margin](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep#results-a-boundary-optimum-at-gamma-005) and well clear of the family's informal \\(\approx5\%\\) "unreliable ranking" threshold. \\(\gamma_{\text{geo}}\\) again clusters tightly: mean \\(0.9627\\), std \\(0.0030\\) — essentially flat across the range, centred about \\(0.02\\) lower than the d=768 mean (\\(0.9810\\)), a small but consistent downward drift with increasing \\(d\\) that also appears at the isotropic-Gaussian d=1024 point elsewhere in this family (\\(\gamma_{\text{geo}} \approx 0.933\\)).
The per-layer residual heatmap below shows the departures from geodesic behaviour concentrating in the middle layers as \\(\gamma\\) increases, peaking sharply around layers 4-7 at \\(\gamma=0.50\\):
Unlike d=768, the recovered intrinsic damping here is not just flat but forms **the cleanest, tightest line of any sweep in the family**:
## The PPL Wiggle Without the Geodesic Wiggle
The PPL curve is **not monotonic** past its minimum — it rises from 244.23 (\\(\gamma=0.05\\)) to a peak at \\(\gamma=0.40\\) (298.05), then **drops back down** at \\(\gamma=0.50\\) (269.77) to nearly the level of \\(\gamma=0.15\\) (264.77). This is the same shape found in the [d=768 sweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep#the-non-monotonic-wiggle-past-the-minimum) (a local peak in the \\(\gamma \approx 0.20\text{-}0.40\\) range, partial recovery at 0.40-0.50) — the same shape appearing at two different widths is a weak argument against pure single-run noise as the sole explanation.
What is new at this width: **the geodesic-residual curve \\(\bar{R}(\gamma)\\) does not share the wiggle.** \\(\bar{R}\\) rises essentially monotonically across the entire swept range (\\(1.21 \to 1.31 \to 1.48 \to 1.52 \to 1.54 \to 1.81 \to 2.04 \to 2.30\\)) — the cleanest \\(\bar{R}\\)-vs-\\(\gamma\\) curve of any sweep in this document, MLP or Gaussian, with no local peak-and-recovery pattern at all. This decouples the two diagnostics in an informative way: whatever is producing the mid-range PPL bump at \\(\gamma \approx 0.20\text{-}0.40\\) (single-seed optimisation noise, or a structural interaction between the anisotropic precision matrices and mid-range explicit friction) is **not** simultaneously producing a mid-range departure from geodesic behaviour — the dynamics stay increasingly non-geodesic in a smooth, monotonic way even where the loss itself wiggles. Disentangling the PPL wiggle's cause would require a 2-3 seed rerun of one or two candidates (e.g. \\(\gamma=0.30\\)-\\(0.40\\)) at one or both widths — not yet done. See [`Determining_optimal_gamma_for_Fock-PARFLM.md` §14.5](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Determining_optimal_gamma_for_Fock-PARFLM.md#145-cross-scale-comparison-a-shared-non-monotonic-signature-at-d768) for the cross-scale discussion (written before this sweep's \\(\bar{R}\\)-vs-PPL decoupling was noted).
## Coincidence, Boundary, and a Second Exact Predictor Match
The two-regime closed-form damping predictor, evaluated at \\(L=16\\), \\(\bar{m}=1.4\\) using the high-\\(d\\) regime constant \\(\rho_{\text{hi}} = 0.565\\), gives the **same prediction regardless of \\(d\\)** within the high-\\(d\\) regime (the formula depends on \\(L\\) and \\(\bar{m}\\), not \\(d\\) directly):
$$
\gamma^{\ast}_{\text{pred}} = \frac{1.4}{16}\ln(1/0.565) = 0.0875 \times 0.571 = 0.050
$$
Empirical \\(\gamma^{\ast} = 0.050\\). **Exact match, zero error.** This is the **second consecutive exact match** at \\(L=16\\) in the high-\\(d\\) regime with the aniso-Gaussian \\(V_\theta\\) (after the [d=768 sweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep#coincidence-boundary-and-an-exact-predictor-match)), and the **third exact match overall** counting the original MLP-\\(V_\theta\\) d=1024, L=16 result. Three independent architectures/widths, one unmodified constant (\\(\rho_{\text{hi}}=0.565\\)), zero free parameters per prediction — the strongest evidence yet that \\(\rho_{\text{hi}}=0.565\\) is a genuine architecture-family invariant at \\(d \gtrsim 768\\).
| Sweep | \\(d\\) | \\(L\\) | \\(V_\theta\\) | \\(\gamma^\ast_{\text{pred}}\\) | \\(\gamma^\ast_{\text{empirical}}\\) | Match |
|---|:---:|:---:|---|:---:|:---:|:---:|
| Original MLP d=1024 sweep | 1024 | 16 | MLP | 0.050 | 0.05 | Exact |
| [d=768 aniso-Gaussian sweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep) | 768 | 16 | aniso-Gaussian + fock-reg | 0.050 | 0.050 | Exact |
| **This sweep** | **1024** | **16** | **aniso-Gaussian + fock-reg** | **0.050** | **0.050** | **Exact** |
This also matches the winner of the original **isotropic**-Gaussian d=1024 partial sweep elsewhere in this family (which only completed 4/8 candidates) — giving **three separate \\(V_\theta\\) variants at this width all agreeing on \\(\gamma=0.05\\)**.
## Comparison Across All Three Widths
| Sweep | \\(\gamma^\ast_{\text{PPL}}\\) | \\(\gamma^\ast_{\bar{R}}\\) | Coincidence? | Optimum shape | Margin vs. runner-up | \\(\bar{R}\\) curve shape |
|---|:---:|:---:|:---:|---|:---:|---|
| [d=384, L=16](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep) | 0.100 | 0.100 | Yes — gap 0 | Interior minimum | large | non-monotonic (\\(\gamma=0.20\\) outlier) |
| [d=768, L=16](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep) | 0.050 | 0.050 | Yes — gap 0 | Boundary minimum | ~4%, modest | non-monotonic wiggle |
| **d=1024, L=16 (this sweep)** | **0.050** | **0.050** | **Yes — gap 0** | **Boundary minimum** | **~10.5%, widest yet** | **monotonic — cleanest in family** |
All three widths show coinciding PPL and geodesic-residual minima, but the pattern strengthens with scale: the margin widens (large at d=384 down to a modest 4% at d=768, then back up to a clean 10.5% at d=1024, this time without the ambiguity of an interior-bowl competing basin), and the geodesic-residual curve gets progressively cleaner — from an outlier-marred non-monotonic curve at d=384, to a genuine-but-unexplained wiggle at d=768, to a fully monotonic curve at d=1024. \\(\gamma_{\text{geo}}\\) clustering also tightens with scale in absolute spread (though drifting slightly downward in mean, \\(0.981 \to 0.963\\) from d=768 to d=1024) — consistent with a model whose intrinsic damping preference becomes both more uniform and slightly lower as width increases.
## Caveats: Short-Sweep Reliability and Two Incomplete Training Logs
This is a **3,000-step, single-seed measurement per gamma**, and the family has one documented case where a short-sweep ranking reversed at full training length: the d=256 TinyStories aniso-Gaussian sweep favoured \\(\gamma=0.150\\) at 3K steps, but the full 20K-step run favoured \\(\gamma=0.300\\) instead. Two considerations favour trusting this sweep's \\(\gamma=0.05\\) recommendation more than that precedent would suggest: (a) this is a **boundary** optimum with \\(\gamma_{\text{geo}}\\) essentially flat across the whole range — there is no competing interior basin to misrank within; and (b) the margin (~10.5%) is the widest of any boundary-optimum sweep in the family, and the predictor's independent, zero-parameter prediction lands on the exact same value.
Two data-quality notes specific to this sweep's raw artifacts:
- **Two training logs are incomplete.** `checkpoints/gamma_0.100/training_log.jsonl` contains only the final (step-3,000) evaluation record, and `checkpoints/gamma_0.200/training_log.jsonl` contains only the final two (step-2,500 and step-3,000) records — both missing the earlier per-500-step entries and the step-2,000 `causal_probe` event that every other candidate's log contains. This looks like a Colab logging/session hiccup specific to those two candidates' runs rather than a training failure: both final checkpoints' `val_ppl` match `sweep_summary.json` and `geodesic_results.json` exactly, and the geodesic residual analysis (which reads directly from the checkpoint, not the training log) completed normally for both. See [Causal-Leak Verification](#causal-leak-verification) below.
- Unlike d=768, this sweep does not show the same non-monotonic pattern in \\(\bar{R}\\) (see [above](#the-ppl-wiggle-without-the-geodesic-wiggle)) — only in PPL — so the usual "watch for the wiggle's instability signature" caveat applies specifically to the loss curve, not the geometric diagnostic, for a full run at this width.
**Recommendation: \\(\gamma=0.05\\) for the d=1024, L=16 aniso-Gaussian full run** — the highest-confidence recommendation of the three widths swept so far in this line.
## How to Get Started
```python
import math, torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")
from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing
from huggingface_hub import hf_hub_download
REPO = "dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep"
GAMMA = "0.050" # the recommended candidate; also available: 0.100, 0.150, 0.200, 0.250, 0.300, 0.400, 0.500
logfreq_path = hf_hub_download(repo_id=REPO, filename="results/logfreq_surprisal_openwebtext.npy")
FORCE_MAX = 2.0 / math.sqrt(1024)
config = FockMultiXiPARFConfig(
vocab_size=50257, d=1024, max_len=1024, L=16,
v_hidden=1024, v_depth=3, dt=1.0,
mass_mode="logfreq", logfreq_path=logfreq_path, logfreq_init_alpha=0.1,
init_gamma=1.0, fixed_gamma=float(GAMMA),
causal_force=True, ln_after_step=True,
xi_channels=5, xi_alpha_inits=[0.50, 0.75, 0.95, 0.99, 0.995],
xi_learnable=True, xi_alpha_init_mode="explicit",
fock_version="v2", n_registers=32,
reverse_channel=True, reverse_channel_stable=True, reverse_channel_pre_ln=True,
reverse_channel_soft_norm=True, reverse_channel_warmup_steps=4000, reverse_channel_per_layer=True,
register_repulsion=True, register_repulsion_coeff=0.05,
prefix_causal_registers=True,
v_phi_kind="structural_competitive", v_phi_n_heads=4, v_phi_d_type=32, v_phi_d_angle=16,
v_phi_eps=0.1, v_phi_phi_hidden=128, v_phi_theta_hidden=128, v_phi_mlp_hidden=128,
top_k=16,
use_output_bias=True, tie_embeddings=False,
score_head_hidden=32,
gumbel_tau_init=1.0, gumbel_tau_min=0.3, gumbel_noise=True,
use_gathered_v_phi=True,
use_layer_checkpoint=True,
ln_before_distance=True, per_layer_v_phi_scale=True,
register_salience_decay=0.5, register_salience_threshold=0.005,
creation_gate_hidden=64, stack_discipline=True,
d_k=64, tau_create_init=8.0,
per_register_tau=True, per_register_keys=True,
ortho_register_init=True,
force_clamp_max=FORCE_MAX,
)
model = FockMultiXiPARFLM(config)
model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
d=1024, K=8, n_ctx=5, n_layers=16, rank=4,
w_scale=1.0,
init_log_precision=-math.log(1024),
precision_max=2.0 / 1024,
force_norm_max=FORCE_MAX,
code_init_std=0.02,
)
install_aniso_depth_routing(model)
ckpt_path = hf_hub_download(repo_id=REPO, filename=f"checkpoints/gamma_{GAMMA}/ckpt_best.pt")
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}") # 363,884,315
print(f"gamma={state['gamma']} step={state['step']:,} val_ppl={state['val_ppl']:.2f}")
```
## Available Artifacts
| Path | Description |
|---|---|
| `checkpoints/gamma_0.050/ckpt_best.pt` ... `checkpoints/gamma_0.500/ckpt_best.pt` | All 8 gamma-sweep checkpoints (step 3,000 each), each a dict with `model_state_dict`, `optimizer_state_dict`, `step`, `val_loss`, `val_ppl`, `gamma`, `v_theta_variant`, `aniso_rank`, `lambda_fock_reg` |
| `checkpoints/gamma_*/training_log.jsonl` | Per-500-step training metrics and the step-2,000 `causal_probe` event, one file per gamma (see [caveats](#caveats-short-sweep-reliability-and-two-incomplete-training-logs) — gamma=0.100 and gamma=0.200 logs are truncated) |
| `sweep_summary.json` | Sweep-level summary (best PPL per gamma, wall-clock) |
| `results/geodesic_results.json` | Full per-gamma geodesic residual data: \\(\bar{R}\\), \\(\gamma_{\text{geo}}\\), excluded fraction, and all 15 per-layer residuals |
| `results/geodesic_overlay_aniso_gaussian_d1024.png` | PPL vs. geodesic residual dual-axis overlay |
| `results/geodesic_per_layer_aniso_gaussian_d1024.png` | Per-layer, per-gamma geodesic residual heatmap |
| `results/gamma_geo_recovery_aniso_gaussian_d1024.png` | Recovered intrinsic damping \\(\gamma_{\text{geo}}\\) vs. training gamma |
| `results/logfreq_surprisal_openwebtext.npy` | Frozen per-token log-frequency (surprisal) lookup used by the `logfreq` mass model |
| `model_aniso_gaussian_vtheta.py` | Anisotropic Gaussian V_theta classes + `install_aniso_depth_routing` (identical file to the TinyStories anchor's and the d=384/d=768 siblings') |
| `config.json` | Full sweep configuration, per-gamma results table, and predictor comparison |
## Training Details
### Training Data
[OpenWebText](https://huggingface.co/datasets/Skylion007/openwebtext), tokenized with GPT-2 BPE (vocab 50257). Each candidate trains on up to 1B tokens (early stopped at 3,000 steps, effective batch 16, block 512 -> ~24.6M tokens actually consumed) and is evaluated on a held-out 2M-token validation slice with no train/val overlap.
### Training Procedure (per gamma candidate)
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW (betas 0.9/0.95, weight decay 0.01) |
| LR schedule | WSD (warmup 0-150, stable 150-1950, decay 1950-3000) |
| Peak learning rate | **1.5e-4** (half of d384/d768's 3e-4) |
| Batch size | 1 x grad-accum 16 (effective 16) |
| Block size | 512 |
| Steps | 3,000 |
| Gradient clipping | per-group (global 1.0; V_phi 0.3 — same as d=384, wider than d=768's 0.5/0.2) |
| Force clamp (model-level and V_theta-level) | \\(2/\sqrt{1024} = 0.0625\\) |
| lambda_V (V_theta regularisation) | 0.01 |
| lambda_fock (coupling regulariser) | 0.005 |
| Seed | 0 |
| Hardware | 1x NVIDIA H100/A100 (Google Colab) |
### Causal-Leak Verification
All 8 checkpoints were trained natively with `prefix_causal_registers=True` from step 0. The bit-exact future-perturbation causal probe passed with `max_delta=0.0` at step 2,000 for **6 of the 8 gamma candidates** with an explicit training-log record. The `gamma=0.100` and `gamma=0.200` training logs are truncated (see [caveats](#caveats-short-sweep-reliability-and-two-incomplete-training-logs)) and do not contain this record, but their final checkpoints match the sweep summary and geodesic-results files exactly, and the architecture and training script are identical to the other 6 candidates that did pass explicitly.
### Training Script
`notebooks/conservative_arch/scaleup/colab_fock_gamma_sweep_geodesic_aniso_gaussian_fockreg_d1024.ipynb` (companion repo) — self-contained Colab notebook that runs the 8-candidate sweep, the geodesic residual analysis, and produces the overlay/heatmap/recovery figures in one pass, requiring no additional training beyond the sweep itself.
## Evaluation Results
### OpenWebText Validation Perplexity (3,000-step sweep candidates)
| \\(\gamma\\) | PPL | Rank |
|---:|---:|:---:|
| **0.050** | **244.23** | 1st |
| 0.150 | 264.77 | 2nd |
| 0.500 | 269.77 | 3rd |
| 0.100 | 270.80 | 4th |
| 0.250 | 284.46 | 5th |
| 0.200 | 288.50 | 6th |
| 0.300 | 295.05 | 7th |
| 0.400 | 298.05 | 8th |
> These PPL values are from 3,000-step short-sweep candidates and are **not comparable** to the fully trained OpenWebText checkpoints elsewhere in this family (e.g. 27.23 PPL after 250K steps for the isotropic d=384 flagship). They exist solely to rank candidate damping coefficients. Note the ranking is not simply increasing with \\(\gamma\\); see [The PPL Wiggle Without the Geodesic Wiggle](#the-ppl-wiggle-without-the-geodesic-wiggle).
See [Results: The Widest Boundary-Optimum Margin Yet](#results-the-widest-boundary-optimum-margin-yet) for the combined PPL / geodesic-residual table.
## SPLM Family Overview
This model is part of the **Semantic Simulation SPLM family**:
| Model | Design | Corpus | PPL | HuggingFace |
|---|---|---|---|---|
| Multi-Xi SPLM (MLP) | Pure scalar potential | TinyStories | 11.51 | [semsimula-splm-multixi](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) |
| Multi-Xi SPLM (SQ3) | Structured scalar potential | TinyStories | 13.33 | [semsimula-splm-multixi-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-splm-multixi-structured-vtheta) |
| Multi-Xi PARFLM (MLP) | Scalar + pairwise forces | TinyStories | 12.06 | [semsimula-parflm-multixi](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi) |
| Multi-Xi PARFLM (SQ3) | Structured scalar + pairwise | TinyStories | 12.27 | [semsimula-parflm-multixi-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi-structured-vtheta) |
| Fock-PARFLM v2.1 (MLP) | PARFLM + Fock registers | TinyStories | 9.70 | [semsimula-fock-parflm](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) |
| Fock-PARFLM v2.1 (SQ3) | Structured + pairwise + Fock | TinyStories | 10.90 | [semsimula-fock-parflm-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-structured-vtheta) |
| Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian) | Bounded multi-context + pairwise + Fock | TinyStories | 16.33 | [semsimula-fock-parflm-depthcond-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-depthcond-vtheta) |
| Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian + fock-reg) | Bounded, ellipsoidal multi-context + pairwise + Fock | TinyStories | 9.04 | [semsimula-fock-parflm-anisogaussian-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, first-order/Fock-G1) | Same as above, gradient-flow integrator | TinyStories | 8.95 | [semsimula-fock-parflm-anisogaussian-vtheta-fock-g1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-fock-g1) |
| Fock Attention (MLP V_theta) | Fock + attention | TinyStories | 9.42 | [semsimula-fock-attention](https://huggingface.co/dimitarpg13/semsimula-fock-attention) |
| Hybrid SPLM+Attn | Attention + SPLM refinement | TinyStories | 8.50 | [semsimula-hybrid-splm](https://huggingface.co/dimitarpg13/semsimula-hybrid-splm) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=384) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 278.27 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=768) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 326.97 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep) |
| **Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=1024)** | **Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis** | **OpenWebText** | **244.23 (best of 8, 3K-step sweep)** | **this repository** |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, Verlet instability, d=384) | Same architecture, two full-run attempts — SCAF stiffness audit identifies structural Verlet instability | OpenWebText | 184.11 / 211.63 (both runs stalled, not final) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability) |
**Collection:** [Semantic Simulation SPLM Model Family](https://huggingface.co/collections/dimitarpg13/semantic-simulation-splm-model-family)
## Bias, Risks, and Limitations
- **Not a final trained model.** Every checkpoint in this repository has been trained for only 3,000 steps — a gamma-selection diagnostic, not a language model intended for generation or downstream use. Do not compare its PPL to fully trained checkpoints elsewhere in this family.
- **Single-seed sweep.** Each gamma candidate is one run; no seed-variance estimate is available. The margin between \\(\gamma=0.05\\) and its nearest competitors (~10-11%) is the widest of any boundary-optimum sweep in this family, mitigating (but not eliminating) this risk; see [Caveats](#caveats-short-sweep-reliability-and-two-incomplete-training-logs).
- **Two training logs are incomplete** (`gamma=0.100`, `gamma=0.200` — missing most per-step entries and the step-2,000 causal-probe record), most likely a Colab logging/session hiccup specific to those runs; the final checkpoints for both are verified consistent with the sweep-summary and geodesic-results files.
- **PPL-vs-gamma wiggle**, shared with the d=768 companion sweep, is not yet explained (single-seed noise vs. a structural property of this \\(V_\theta\\) configuration) — see [The PPL Wiggle Without the Geodesic Wiggle](#the-ppl-wiggle-without-the-geodesic-wiggle). Notably the geodesic-residual curve does not share this wiggle at this width.
- **OpenWebText only, English only.** No instruction tuning, no RLHF/DPO, no safety filtering.
- **Geodesic residual is a diagnostic, not a training objective.** The model was trained to minimise cross-entropy; the near-geodesic behaviour at \\(\gamma=0.05\\) is an emergent structural property, not something the loss function directly optimises for.
- **Very large \\(V_\theta\\) hypernetwork.** At 252.0M parameters, \\(V_\theta\\) alone is 69% of the 363.9M-parameter total — the largest \\(V_\theta\\) share of any model in this family, driven by the low-rank correction \\(B_k \in \mathbb{R}^{d \times r}\\) scaling with \\(d\\).
- **No causal-leak issue.** All 8 checkpoints were trained natively with `prefix_causal_registers=True`; see [Causal-Leak Verification](#causal-leak-verification).
## Citation
```bibtex
@misc{Gueorguiev2026SemSim,
author = {Gueorguiev, Dimitar P.},
title = {Semantic Simulation: A Prescriptive Lagrangian Framework
for Efficient Semantic Inference --- A Conservative-by-
Construction Language Model and the Shared-Potential
Separator, with a Correspondence to Joint Embedding
Predictive Architectures},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.19712427},
url = {https://doi.org/10.5281/zenodo.19712427},
note = {Companion code repository:
\url{https://github.com/dimitarpg13/semsimula-paper}}
}
```
## Environmental Impact
- **Hardware:** 1x NVIDIA H100/A100 80GB (Google Colab)
- **Training:** 8 x 3,000 steps = 24,000 total training steps across the sweep, plus inference-only geodesic residual analysis (10 validation batches x 8 checkpoints, no additional training)
- **Carbon footprint:** small; a single-GPU research sweep, estimated on the order of a few kg CO2