dimitarpg13/semsimula-fock-parflm-depthcond-vtheta-openwebtext

⚠️ SUPERSEDED — this model-experiment is deprecated and no longer maintained

This checkpoint is the isotropic-Gaussian $V_{\theta }V\_\backslash theta$ OpenWebText flagship at d=384. It is being superseded by the anisotropic Gaussian $V_{\theta }V\_\backslash theta$ with Fock-coupling regularisation line, a materially different architecture (ellipsoidal, depth-conditioned, low-rank precision wells, plus a log-barrier coupling regulariser — see the TinyStories anchor for the analytical form) that has decisively outperformed the isotropic variant at every scale tested so far.

The OpenWebText-scale successor is being rolled out gamma-sweep-first, then full-run:

Scale	Gamma sweep (diagnostic)	Full run
d=384, L=16	semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep	in progress at $\gamma =0.10\backslash gamma=0.10$
d=768, L=16	semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep	not yet launched
d=1024, L=16	semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep	not yet launched

This repository is being kept online rather than deleted purely for link continuity: several sibling model cards in the Semantic Simulation SPLM family reference it directly (family-overview tables, and the geodesic-residual cross-scale comparisons in the d384/d768/d1024 gamma-sweep cards above). It has been removed from the collection's front page to avoid steering new readers here. All content below this notice (including the causal-leak disclaimer) is preserved unmodified as the historical record of this line of experiments.

---

CAUSAL-LEAK INVESTIGATION IN PROGRESS — REPORTED PPL IS UNDER REVIEW

A post-publication causal audit has identified an architectural causal leak in the Fock reverse-channel mechanism used by this model. The reverse channel blends each token's content into a global register state that is shared across all positions within the same integration step. Because this shared state is read by every token — including tokens earlier in the sequence — future-token information leaks backward into past-token predictions, violating the strict left-to-right causality required for valid autoregressive perplexity.

Key findings from the causal probe (on a later Phase-3 checkpoint from the identical architecture):

Metric	Value
PPLstandard (leaky, as reported)	7.69
PPLhonest (leak-free, target-relocated)	258.07
NLL inflation	+3.51 nats
PPL inflation factor	~33×

The Phase-3 probe was run on a checkpoint at step 103,500 that reached a reported PPL of 9.50. This Phase-2 checkpoint (27.23 PPL) uses the same leaky architecture — the reverse channel is active (reverse_channel_scale > 0) and the register state is not prefix-causal. The honest PPL of this specific checkpoint has not yet been measured, but the leak is present by construction.

What this means:

• The 27.23 PPL should not be cited as a validated perplexity result.
• The model's true autoregressive quality is substantially worse than reported.
• The reverse-channel ablation (27.23 vs 125.94) remains a valid relative comparison (both arms share the same leak architecture), but both baselines are inflated.
• All architectural contributions (depth-conditioned V_θ, Fock mechanism, geodesic analysis) remain valid — only the absolute PPL numbers are affected.

Status: An architectural fix (prefix_causal_registers) has been implemented and verified to produce exact 0.0 future-perturbation sensitivity in float64. Re-training with the fixed architecture is required to obtain honest PPL numbers. This model card will be updated with corrected results once the re-run is complete.

For full details, see the companion audit: Fock-PARFLM_Causal_Leak_Audit_Results.md.

[SUPERSEDED] Fock-PARFLM v2.1 with Depth-Conditioned Multi-Context Gaussian V_theta (OpenWebText, 27.23 PPL — under causal-leak review)

The OpenWebText-scale flagship of the Fock-PARFLM conservative-language-model line. It replaces the MLP scalar potential V_θ with a depth-conditioned, multi-context bank of Gaussian wells (5 temporal-context heads × 8 wells = 40 attractors), keeps the sparse pairwise potential V_φ and the full Fock register mechanism (32 registers, LIFO stack discipline, reverse channel), and trains it on OpenWebText to a validation perplexity of 27.23 using only ~2B training tokens.

Note: The 27.23 PPL was measured with the original (non-prefix-causal) architecture and is subject to the causal leak described above. See the disclaimer for details.

This is a from-scratch, attention-free language model: every token's hidden state is evolved by a damped second-order dynamical system whose force is minus the gradient of a learned scalar potential, plus a sparse pairwise force, plus Fock creation/destruction/exchange forces. There is no softmax attention anywhere in the model.

• 53.4M parameters (GPT-2-Small class), d = 384, L = 16 integration steps.
• 27.23 PPL on OpenWebText after 100K + 150K steps (graduated 1B → 2B tokens). ⚠️ Under causal-leak review.
• Two design decisions were decisive for reaching this number: the Fock reverse channel (removing it plateaus the model above 125 PPL — see the ablation below) and untied input/output embeddings (tied embeddings were unstable at this depth).

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Model
• Architecture
• What Made 27 PPL: Settings and Optimizations
• The Reverse-Channel Ablation
• Geodesic Preservation: Riemannian Structure in the Dynamics
• How to Get Started
• Reproducing the Result
• Training Details
• Evaluation Results
• Available Artifacts
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation

When to Use This Model

This is a research checkpoint demonstrating that a conservative, attention-free, physics-informed language model can reach GPT-2-Small-class perplexity on a general web corpus. Choose it when you want to:

• study a non-transformer sequence model with an explicit, interpretable force decomposition (self-energy V_θ, pair energy V_φ, Fock registers);
• read explicit semantic attractors (40 Gaussian well centres, conditioned per temporal horizon and per layer) directly from the parameters;
• experiment with constant-memory inference dynamics (the state is integrated step-by-step; there is no growing KV cache from a softmax attention).

It is not a general-purpose chat or instruction model (see Limitations).

⚠️ Causal-leak caveat: Due to the identified causal leak in the reverse channel, this checkpoint's perplexity is inflated (optimistically low). Use the architecture and code for research purposes, but do not rely on the reported 27.23 PPL as a validated benchmark.

⚠️ Superseded caveat: For new work at this scale, prefer the anisotropic Gaussian $V_{\theta }V\_\backslash theta$ + Fock-reg line linked in the notice at the top of this card — it has decisively outperformed this isotropic variant at every scale tested (TinyStories and, via the gamma-sweep diagnostics, OpenWebText). This card is kept for historical reference and link continuity only.

Architecture

Input tokens x_1, ..., x_T
       |
   Untied token embedding E[x] + learned positional P[t]
       |
   For each of L=16 damped-Euler integration steps (shared force field):
       |
       +-- K=5 causal-EMA context channels:
       |     xi^(m)_t = causal_ema(h, alpha_m)      [horizons ~2 .. ~200 tokens]
       |
       +-- Depth-conditioned multi-context V_theta (Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)            [per-layer depth code e_g]
       |     V_theta  = sum_m  Gaussian_well_bank_m(xi_g^(m), h)  [5 heads x 8 wells]
       |     f_theta  = -grad_h V_theta             [conservative self-force]
       |
       +-- Sparse pairwise V_phi (structural-competitive, 4 heads):
       |     top-k=16 past tokens per query (Gumbel routing)
       |     f_phi = -grad_h V_phi                  [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=32 virtual registers, Q/K/V creation gates, d_k=64
       |     LIFO stack discipline, per-register tau/keys, ortho init
       |     register repulsion (Gram penalty, lambda=0.05)
       |     reverse channel (per-layer, stabilised, warmup 4000)  [non-conservative]
       |     *** CAUSAL LEAK SOURCE: register state is shared across all positions ***
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       +-- LayerNorm(h)
       |
   Logits = h @ W_out^T + b_out          [UNTIED W_out; unigram-init bias b_out]

The mass m is a frozen per-token log-frequency (surprisal) lookup; the damping γ is a learned scalar fixed at 0.30 for this run. Note that the effective damping is much smaller than the nominal γ: the LayerNorm applied after each integration step rescales the hidden state, absorbing most of the velocity decay, so the dynamics are heavily underdamped even at γ = 0.30. The per-step force is minus the gradient of a scalar potential (plus the sparse pair force and the Fock forces), so each step is conservative-by-construction except for the single explicitly non-conservative reverse channel.

Parameter	Value
Hidden dim (d)	384
Integration steps (L)	16
Max sequence length	1024 (trained at block 512)
Vocab (GPT-2 BPE)	50257
V_theta kind	depth-conditioned multi-context Gaussian
V_theta heads × wells	5 × 8 = 40 attractors
Depth codes	per-layer (L × n_ctx × d), init std 0.02
Xi channels (K)	5
Xi alpha inits	0.50, 0.75, 0.95, 0.99, 0.995
V_phi kind	structural-competitive
V_phi heads	4
V_phi d_type / d_angle	32 / 16
Top-k (sparse routing)	16
Fock version	v2.1
Registers (M)	32
Register d_k	64
Stack discipline	LIFO
Per-register tau / keys	Yes
Reverse channel	Yes (per-layer, stabilised) — causal leak source
Register repulsion	Gram, lambda 0.05
Mass model	logfreq (frozen surprisal)
Damping gamma	0.30 nominal, ~0.016 effective (LayerNorm prevents compounding; see geodesic analysis)
Embeddings	UNTIED (separate W_out) + unigram bias
lambda_V (V_theta regulariser)	0.01
Prefix-causal registers	No (leak present; fix available but requires re-training)
Total parameters	53,378,075

What Made 27 PPL: Settings and Optimizations

The self-contained notebooks in this repo (they pull the model code from this repo's code/ folder) reproduce the number. The settings that moved the needle, briefly:

1. Depth-conditioned multi-context V_theta. A single shared Gaussian well bank is reused across all L layers, but each layer adds a small learned depth code to the context before the bank projections. This gives each integration step a distinct effective potential at a cost of only ~L·n_ctx·d ≈ 31k parameters, instead of ~178M for fully untying the bank. Each of the 5 xi channels gets its own bank so attractors are conditioned on a specific temporal horizon rather than a blurred concatenation.
2. Fock reverse channel (stabilised, per-layer). The one non-conservative force. It is the single most important component for perplexity (see the ablation below), but it was also the dominant gradient-spike source. It is bounded with QK-norm, a soft-floored output RMS-norm, a 4000-step gate warmup, and a per-layer gate (rather than one global scalar) so its gradient does not aggregate into a high-variance knob. ⚠️ This component is also the sole carrier of the causal leak: the reverse channel blends token content into a shared register state that is read by all positions, allowing future information to leak backward.
3. Untied embeddings + unigram output bias. Tied embeddings were unstable at d=384 with this depth; untying the output head and initialising its bias from the log-unigram frequency fixed the long tail and stabilised training.
4. WSD (Warmup–Stable–Decay) learning-rate schedule. Peak LR is held through the middle of training so the wells have time to separate and specialise before the final cosine anneal. The 27.23 result comes from a graduated two-phase run: phase 1 (100K steps, 1B tokens) reaches ~63.7 PPL; a phase-2 continuation (150K steps, 2B tokens) with a fresh WSD cycle drives it to 27.23.
5. Wider routing. top-k doubled to 16, V_phi type/angle subspaces doubled to 32/16, and 4 V_phi heads — cheap in parameters, larger directional-force diversity.
6. Per-group gradient clipping + watchdog. Each top-level module is clipped to its own max-norm (V_phi and Fock gates to 0.3/0.1), with an EMA watchdog that reloads the last good checkpoint on sustained grad-norm blow-ups. This tames the second-order (create_graph=True) gradient cascade that appears late in training.
7. Register repulsion. A Gram-matrix penalty (lambda 0.05) keeps the 32 registers from collapsing onto a few effective directions.

The Reverse-Channel Ablation

The reverse channel is the sole non-conservative force in the model. Training the identical architecture with the reverse channel disabled (arm e5a) never breaks 120 PPL and plateaus around 125.94 PPL; with the stabilised reverse channel on (arm e5c, this model) it reaches 27.23 PPL.

Arm	Reverse channel	Best val PPL	Outcome
e5c (this model)	on (stabilised, per-layer)	27.23	continues improving through 250K steps
e5a	off	125.94	plateaus early

The gap is not marginal — it is the difference between a working model and a stuck one. A purely conservative force field (all forces equal to minus a gradient of a potential) is measurably insufficient here; a bounded amount of directed, non-conservative information routing is what lets the dynamics match what attention provides. This is discussed at length in the companion note Fock_Mechanism_Ablation_Study_d384_OpenWebText.md.

⚠️ Causal-leak note: The reverse channel's dramatic impact on PPL (125.94 → 27.23) is now understood to be partly — and possibly largely — attributable to the causal leak it introduces. The reverse channel allows each token to read a register state that summarises all tokens in the window, including future ones. This is architecturally equivalent to giving the model access to bidirectional context for its predictions, which trivially lowers cross-entropy. The ablation remains valid as a relative comparison (arm e5a has no leak because it has no reverse channel), but the absolute PPL numbers in both arms are affected: arm e5c's 27.23 is leak-inflated (optimistically low), while arm e5a's 125.94 is a legitimate (leak-free) autoregressive perplexity. The true gap between "reverse channel on" and "reverse channel off" will only be known after re-training with prefix_causal_registers=True.

Geodesic Preservation: Riemannian Structure in the Dynamics

A defining claim of the Semantic Simulation framework is that the hidden-state trajectories are not arbitrary learned curves but approximate Riemannian geodesics on a manifold whose metric is induced by the learned potentials. We test this by computing the damped geodesic residual across a sweep of the damping coefficient.

The experiment

For each candidate damping value, we train a fresh d=384 L=16 model for 3,000 steps, then measure the per-layer geodesic residual

$$R_{\mathrm{\ell }}=\frac{\parallel h_{\mathrm{\ell }+1}-h_{\mathrm{\ell }}-\mathrm{\Delta }t\cdot v_{\mathrm{\ell }}\parallel }{\parallel h_{\mathrm{\ell }+1}-h_{\mathrm{\ell }}\parallel +ϵ}R\_\backslash ell\; =\; \backslash frac\{\backslash lVert\; h\_\{\backslash ell+1\}\; -\; h\_\backslash ell\; -\; \backslash Delta\; t\; \backslash cdot\; v\_\backslash ell\; \backslash rVert\}\{\backslash lVert\; h\_\{\backslash ell+1\}\; -\; h\_\backslash ell\; \backslash rVert\; +\; \backslash epsilon\}$$

where the numerator is the deviation of the actual layer-step from the velocity-predicted geodesic step. Averaging across layers and validation batches gives the mean residual R̄(γ). If R̄ = 1 the dynamics are perfectly metric-compatible; values above 1 indicate the force field pushes the trajectory away from the geodesic.

Results

The figure below overlays validation perplexity (blue, left axis) and the damped geodesic residual R̄ (red, right axis) across eight gamma values. Two key findings emerge:

1. Near-geodesic at low damping. At the geodesic-optimal γ = 0.05, the residual is R̄ = 1.041 — the dynamics deviate from the metric-compatible geodesic by only 4% per layer on average. This confirms that the learned force field produces trajectories that are close to geodesics on the manifold it defines.

2. PPL-geodesic coincidence breaks down at d=384. The PPL-optimal γ is 0.25 (342 PPL), while the geodesic-optimal γ is 0.05 (R̄ = 1.041). At larger hidden dimensions (d=768, d=1024), the PPL and geodesic minima coincide at γ = 0.05. The d=384 breakdown is a dimension-dependent phase transition: smaller models prefer stronger damping for perplexity, even though it distorts the geodesic structure.

γ	γeff	Val PPL	Geodesic residual (R̄)	γgeo	Notes
0.05	0.003	484	1.041	0.917	geodesic-optimal
0.10	0.006	419	1.050	0.880
0.15	0.009	350	1.309	0.974
0.20	0.011	397	1.345	0.871
0.25	0.014	342	1.418	0.988	PPL-optimal
0.30	0.016	354	2.115	0.995	this model's training gamma
0.40	0.021	741	1.636	0.942
0.50	0.026	370	1.758	0.910

γ_geo is the recovered intrinsic damping — the γ value that would minimise R̄ if the residual were re-evaluated against this specific trajectory rather than the nominal training γ. It clusters tightly at 0.87–1.00 across the entire sweep, essentially independent of the nominal training γ.

The per-layer heatmap reveals that geodesic violation concentrates in the middle layers (layers 6-10) at high damping, while early and late layers stay near-geodesic — consistent with the middle layers carrying the heaviest semantic-routing burden.

Note on γ=0.30 (this model's training gamma). The R̄=2.115 at γ=0.30 is the highest in the sweep — a non-monotonic spike above both neighbors (γ=0.25: 1.418, γ=0.40: 1.636). This does not mean the model has left the Riemannian manifold or stopped following geodesics. The residual tests compliance with the damped geodesic equation at the nominal γ; a large R̄ means the nominal hyperparameter is a poor match for the trajectory's intrinsic damping, not that the trajectory is non-geodesic. The γ_geo column recovers the damping the trajectory actually exhibits: for this checkpoint γ_geo ≈ 0.995 (near-critically-damped). If R̄ were re-evaluated at γ_eval = γ_geo, it would drop back toward R̄ ≈ 1. The model follows a damped geodesic at its own intrinsic damping regardless of the training hyperparameter — and that is all the geometric toolkit needs. Both the energy-dissipation hallucination detector and native chain-of-thought via Fock register dynamics require only a valid Jacobi metric (guaranteed by the exact potential V_θ) and compliance with some damped geodesic equation, not that the nominal γ matches the intrinsic one.

Interpretation

The fact that R̄ ≈ 1 at low damping is not a trivial outcome. The model was trained to minimise cross-entropy, not to produce geodesic trajectories. That the learned force field happens to be nearly metric-compatible is an emergent structural property — the scalar and pairwise potentials organise the hidden-state manifold so that the minimum-energy paths are also the most predictive paths. This is the central prediction of the Lagrangian framework: conservative forces should produce geodesic-like dynamics because the Euler-Lagrange equations of a conservative system are the geodesic equations of the kinetic-energy metric.

An important caveat when interpreting the nominal γ values: the LayerNorm after each integration step rescales the hidden state, preventing the per-step velocity decay from compounding across layers. Without LayerNorm, the velocity after L = 16 steps would be reduced to v₀ / (1 + γ)^L — at γ = 0.30, this means only 1.5% of the initial velocity survives, a catastrophically overdamped regime. With LayerNorm resetting the state scale each step, the decay does not compound, and the total velocity reduction over L layers is closer to a single-step reduction.

The effective damping γ_eff (the γ_eff column in the table above) is estimated as the decompounded per-step equivalent:

$${\gamma }_{\text{eff}}=(1+\gamma {)}^{1\mathrm{/}L}-1\approx \gamma \mathrm{/}L\backslash gamma\_\{\backslash text\{eff\}\}\; =\; (1\; +\; \backslash gamma)^\{1/L\}\; -\; 1\; \backslash approx\; \backslash gamma\; /\; L$$

At γ = 0.30 with L = 16, this gives γ_eff ≈ 0.016 — an 18× reduction from the nominal value. Even the PPL-optimal γ = 0.25 corresponds to γ_eff ≈ 0.014. The model operates in a heavily underdamped regime across the entire sweep range, which explains why d=384 can tolerate nominally high γ values without over-dissipating. At larger hidden dimensions (d=768, d=1024), the LayerNorm correction is relatively weaker, and the nominal and effective damping converge, which is consistent with the PPL-geodesic coincidence reappearing at γ = 0.05.

The geodesic analysis data and images are in the geodesic/ folder. For the full cross-scale analysis (d=384, d=768, d=1024), see the companion note Geodesic_Preservation_Experiment.md.

How to Get Started

The full model-code closure is bundled in this repo under code/, so no external clone is needed.

import sys, torch
from pathlib import Path
from huggingface_hub import snapshot_download, hf_hub_download

REPO = "dimitarpg13/semsimula-fock-parflm-depthcond-vtheta-openwebtext"
BEST = "checkpoints/fock_dcvt_owt_xi5long_topk16_dt32da16_mh4_dcvt5x8_ob_untied_wsd_e5c_plgate_rep0.05_best.pt"

# 1. Download the bundled model code and put it on sys.path
code = Path(snapshot_download(repo_id=REPO, allow_patterns=["code/**"])) / "code"
for sub in ["", "parf", "multixi", "sarf_mass_variant", "energetic_minima"]:
    sys.path.insert(0, str(code / sub) if sub else str(code))

from model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from model_gaussian_vtheta import DepthConditionedMultiContextGaussianVTheta, install_depth_routing

# 2. Load the best checkpoint (it carries the exact model_cfg it was built with)
state = torch.load(hf_hub_download(repo_id=REPO, filename=BEST),
                   map_location="cpu", weights_only=False)

# 3. Point the mass model at the hosted surprisal table, then rebuild the config.
#    (The model reads logfreq_path at construction; the saved path is a Colab path.)
mc = dict(state["model_cfg"])
mc["logfreq_path"] = hf_hub_download(repo_id=REPO,
                                     filename="data/logfreq_surprisal_openwebtext.npy")
cfg = FockMultiXiPARFConfig(**mc)
model = FockMultiXiPARFLM(cfg)

# 4. Swap in the depth-conditioned multi-context Gaussian V_theta and load weights.
model.V_theta = DepthConditionedMultiContextGaussianVTheta(
    d=cfg.d, K=8, n_ctx=5, n_layers=cfg.L,
)
install_depth_routing(model)
model.load_state_dict(state["model_state_dict"], strict=False)
model.eval()

print(f"Parameters: {model.num_params():,}")            # 53,378,075
print(f"Checkpoint step {state['step']:,}  val PPL {state['val_ppl']:.2f}")  # step 150,000  PPL 27.23

The exact V_theta construction (well count, precision init, depth-code std) matches the training notebooks; the simplest way to rebuild the model identically is to run the model-config and V_theta-swap cells of notebooks/colab_fock_depthcond_vtheta_openwebtext.ipynb, which construct it from the same model_cfg fields and then load the checkpoint.

Reproducing the Result

Two self-contained Colab notebooks are included. Both pull all model code from this HF repo (no GitHub clone) and download OpenWebText from the Hub.

Notebook	Role
notebooks/colab_fock_depthcond_vtheta_openwebtext.ipynb	Phase 1 — 100K steps, 1B tokens, WSD, reaches ~63.7 PPL
notebooks/colab_fock_depthcond_vtheta_openwebtext_ext.ipynb	Phase 2 — 150K-step continuation, 2B tokens, fresh WSD cycle, reaches 27.23 PPL

Run phase 1 first, then phase 2. The extension notebook will automatically seed from the hosted phase-1 checkpoint (checkpoints/..._step99000_best.pt) if no local/Drive checkpoint is present, so phase 2 can also be run on its own.

⚠️ Note: These notebooks use the original (leaky) architecture. To reproduce with the causal-leak fix, set prefix_causal_registers=True in the FockMultiXiPARFConfig. See the companion audit for the updated training notebooks.

Training Details

Training Data

OpenWebText — an open reproduction of the WebText corpus, tokenized with GPT-2 BPE (vocab 50257). Training consumed a graduated token budget (1B tokens in phase 1, 2B tokens in phase 2; ~2.05B tokens total).

Validation set (held-out, no data leakage). When OpenWebText is first streamed and tokenised, the code requests MAX_TRAIN_TOKENS + 2,000,000 total tokens. The last 2M tokens of the stream are sliced off as the validation set; the first N tokens become the training set. The two are cached as separate files (openwebtext_val_2M.npy and openwebtext_train_{N}M.npy) with zero token-level overlap. When the token budget graduated from 1B to 2B, the training pool grew but the validation set remained the same 2M-token held-out slice, so all PPL numbers across phases are directly comparable. Each evaluation step draws 5 random batches of length 512 from the validation set, computes mean cross-entropy loss, and exponentiates: PPL = exp(L̄_val). The validation file is included in this repo at data/openwebtext_val_2M.npy.

Training Procedure

Hyperparameter	Value
Optimizer	AdamW
Peak learning rate	1.5e-4 (phase 2), 3e-4 (phase 1)
LR schedule	WSD (warmup–stable–decay), floor 1.5e-5
Warmup	2,000 steps
Weight decay	0.01
Effective batch	16 (batch 8 × grad-accum 2)
Block size	512
Global grad clip	1.0 (per-group; V_phi 0.3, Fock gates 0.1)
lambda_V (V_theta regulariser)	0.01
Register repulsion coeff	0.05
Steps	100,000 (phase 1) + 150,000 (phase 2)
Seed	0
Hardware	1× NVIDIA H100 80GB (Google Colab)

Memory-critical optimizations enabled during training: gradient (layer) checkpointing over the integration steps, a gathered sparse-V_φ path, and LayerNorm-before-distance in the routing scorer.

Evaluation Results

OpenWebText Validation Perplexity

Model	PPL	Params	Reverse channel	Causal leak
Fock-PARFLM v2.1 (depth-cond multi-context Gaussian, this model)	27.23 ⚠️	53.4M	on	Yes — under review
Fock-PARFLM v2.1, reverse channel off (ablation arm e5a)	125.94	53.4M	off	No (no reverse channel)

⚠️ The 27.23 PPL is measured with the original architecture which has a confirmed causal leak via the reverse channel's shared register state. The honest (leak-free) PPL of this checkpoint is expected to be substantially higher. The ablation arm e5a (reverse channel off) is leak-free and its 125.94 PPL is a valid autoregressive perplexity. See the causal-leak disclaimer at the top of this card.

PPL is evaluated on a held-out 2M-token validation slice of OpenWebText (see Training Data for the split protocol). There is no overlap between training and validation tokens. PPL here is on OpenWebText, a general web corpus. It is not comparable to the TinyStories perplexities reported for the smaller SPLM-family cards (TinyStories is a far easier, synthetic corpus).

Best checkpoints over training

Checkpoint	Step	Val PPL	Causal leak status
..._best.pt (= step 150000)	150,000 (phase 2)	27.23 ⚠️	Leak present
..._step140500_best.pt	140,500	27.90 ⚠️	Leak present
..._step135000_best.pt	135,000	29.00 ⚠️	Leak present
..._step99000_best.pt	99,000 (phase-1 end)	63.69 ⚠️	Leak present

Learned Xi-Channel Decay Rates

The final learned alpha values are [0.172, 0.322, 0.498, 0.639, 0.965] (init [0.50, 0.75, 0.95, 0.99, 0.995]), spreading the five causal-EMA channels across short-to-long temporal horizons.

Available Artifacts

Path	Description
checkpoints/..._best.pt	Best checkpoint (27.23 PPL ⚠️, step 150000) — identical to step150000
checkpoints/..._step140500_best.pt	27.90 PPL milestone
checkpoints/..._step135000_best.pt	29.00 PPL milestone
checkpoints/..._step99000_best.pt	End of phase 1 (63.69 PPL); seeds the extension notebook
code/	Full self-contained model-code closure (parf, multixi, sarf_mass_variant, energetic_minima, data_module)
notebooks/colab_fock_depthcond_vtheta_openwebtext.ipynb	Phase-1 training notebook
notebooks/colab_fock_depthcond_vtheta_openwebtext_ext.ipynb	Phase-2 extension notebook
data/openwebtext_val_2M.npy	Held-out validation tokens
data/logfreq_surprisal_openwebtext.npy	Frozen per-token surprisal (mass model)
results/training_log_phase1.jsonl	Phase-1 per-step metrics
results/training_log_phase1_phase2.jsonl	Full 250K-step per-step metrics
geodesic/geodesic_overlay_d384.png	PPL vs geodesic residual overlay plot
geodesic/geodesic_per_layer_d384.png	Per-layer geodesic residual heatmap
geodesic/gamma_geo_recovery_d384.png	Recovered intrinsic damping across gamma
geodesic/geodesic_results.json	Full per-gamma, per-layer geodesic residual data
config.json	Model + training configuration and headline results

Each checkpoint is a dict with model_state_dict, optimizer_state_dict, model_cfg, train_cfg, step, val_ppl, val_loss, and the learned xi_alphas.

SPLM Family Overview

This model is (was) part of the Semantic Simulation SPLM family; it has been removed from the collection's front page as superseded (see the notice at the top of this card) but the table below is left as it was at publication time, for historical reference. The other cards are smaller TinyStories models; this was the first OpenWebText-scale member.

Model	Design	Corpus	HuggingFace
Multi-Xi SPLM	Pure scalar potential	TinyStories	semsimula-splm-multixi
Multi-Xi PARFLM	Scalar + pairwise	TinyStories	semsimula-parflm-multixi
Fock-PARFLM v2.1 (MLP)	+ Fock registers	TinyStories	semsimula-fock-parflm
Fock-PARFLM v2.1 (SQ3)	+ structured V_theta	TinyStories	semsimula-fock-parflm-structured-vtheta
Fock-Attention	Fock + attention	TinyStories	semsimula-fock-attention
Hybrid SPLM+Attn	Attention + SPLM refinement	TinyStories	semsimula-hybrid-splm
[SUPERSEDED] Fock-PARFLM v2.1 (depth-cond multi-context Gaussian)	+ depth-conditioned Gaussian V_theta	OpenWebText	this model

For the current OpenWebText-scale line (anisotropic Gaussian $V_{\theta }V\_\backslash theta$ + Fock-reg), see the superseded notice at the very top of this card.

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Research checkpoint only. A proof-of-concept that a conservative, attention-free dynamical language model can reach GPT-2-Small-class perplexity on a general corpus — not a production system.
• OpenWebText only, English only. No instruction tuning, no RLHF/DPO, no safety filtering. Generations reflect the biases of web text.
• Small scale. 53.4M parameters, d=384.
• Training cost of the force field. The self-force f_θ = -∇_hV_θ and the Fock forces are computed with a second-order graph (create_graph=True); this is the source of the late-training gradient spikes the watchdog and per-group clipping are designed to catch. See the companion note Training_Instabilities_in_Fock-PARFLM_with_structured_V_theta.md.
• Reverse channel is load-bearing. The single non-conservative force cannot be removed without losing most of the model's quality (125.94 vs 27.23 PPL).
• ⚠️ Confirmed causal leak in the reverse channel. The Fock reverse channel introduces a causal leak by blending each token's content into a global register state shared across all sequence positions within the same integration step. Future-token information flows backward into past-token predictions, violating autoregressive causality. A post-publication causal probe on a later checkpoint from the same architecture measured +3.51 nats of NLL inflation (~33× PPL factor). An architectural fix (prefix_causal_registers) has been implemented and verified, but this checkpoint was trained without it. The reported 27.23 PPL is not a valid autoregressive perplexity — re-training with the fix is required to obtain honest numbers. See the full audit.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Companion code repository:
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: 1× NVIDIA H100 80GB (Google Colab)
• Training: 250,000 steps total (100K + 150K), ~2.05B tokens
• Carbon footprint: small; a single-GPU research run, estimated on the order of a few kg CO2