--- pipeline_tag: text-generation library_name: pytorch language: - en license: cc-by-4.0 tags: - research - semsimula - conservative-language-model - scalar-potential - lagrangian-mechanics - energy-based-model - physics-informed - fock-space - parflm - parf - multi-channel-xi - depth-conditioned-vtheta - multi-context-vtheta - gaussian-mixture-potential - structured-vtheta - virtual-registers - reverse-channel - controlled-conservativity - sparse-routing - wsd-schedule - untied-embeddings - openwebtext - non-transformer - attention-free - constant-memory-inference - riemannian-geometry - riemannian-geodesics datasets: - Skylion007/openwebtext metrics: - perplexity model-index: - name: semsimula-fock-parflm-depthcond-vtheta-openwebtext results: - task: type: text-generation dataset: type: Skylion007/openwebtext name: OpenWebText split: validation metrics: - type: perplexity value: 27.23 name: Validation Perplexity verified: false --- > **⚠️ SUPERSEDED — this model-experiment is deprecated and no longer maintained** > > This checkpoint is the **isotropic**-Gaussian \\(V_\theta\\) OpenWebText flagship at d=384. It is being superseded by the **anisotropic** Gaussian \\(V_\theta\\) with Fock-coupling regularisation line, a materially different architecture (ellipsoidal, depth-conditioned, low-rank precision wells, plus a log-barrier coupling regulariser — see the [TinyStories anchor](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta) for the analytical form) that has decisively outperformed the isotropic variant at every scale tested so far. > > The OpenWebText-scale successor is being rolled out gamma-sweep-first, then full-run: > > | Scale | Gamma sweep (diagnostic) | Full run | > |---|---|---| > | d=384, L=16 | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep) | in progress at \\(\gamma=0.10\\) | > | d=768, L=16 | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep) | not yet launched | > | d=1024, L=16 | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep) | not yet launched | > > This repository is being **kept online rather than deleted** purely for link continuity: several sibling model cards in the [Semantic Simulation SPLM family](https://huggingface.co/collections/dimitarpg13/semantic-simulation-splm-model-family) reference it directly (family-overview tables, and the geodesic-residual cross-scale comparisons in the d384/d768/d1024 gamma-sweep cards above). It has been **removed from the collection's front page** to avoid steering new readers here. All content below this notice (including the causal-leak disclaimer) is preserved unmodified as the historical record of this line of experiments. --- > **CAUSAL-LEAK INVESTIGATION IN PROGRESS — REPORTED PPL IS UNDER REVIEW** > > A post-publication causal audit has identified an **architectural causal leak** in the Fock reverse-channel mechanism used by this model. > The reverse channel blends each token's content into a **global register state that is shared across all positions within the same integration step**. > Because this shared state is read by every token — including tokens *earlier* in the sequence — future-token information leaks backward into past-token predictions, > violating the strict left-to-right causality required for valid autoregressive perplexity. > > **Key findings from the causal probe (on a later Phase-3 checkpoint from the identical architecture):** > > | Metric | Value | > |---|---| > | PPLstandard (leaky, as reported) | 7.69 | > | PPLhonest (leak-free, target-relocated) | 258.07 | > | NLL inflation | **+3.51 nats** | > | PPL inflation factor | **~33×** | > > The Phase-3 probe was run on a checkpoint at step 103,500 that reached a reported PPL of 9.50. > **This Phase-2 checkpoint (27.23 PPL) uses the same leaky architecture** — the reverse channel is active (`reverse_channel_scale` > 0) and the register state is *not* prefix-causal. > The honest PPL of this specific checkpoint has not yet been measured, but the leak is present by construction. > > **What this means:** > - The **27.23 PPL should not be cited as a validated perplexity result**. > - The model's true autoregressive quality is substantially worse than reported. > - The reverse-channel ablation (27.23 vs 125.94) remains a valid *relative* comparison (both arms share the same leak architecture), but both baselines are inflated. > - All architectural contributions (depth-conditioned Vθ, Fock mechanism, geodesic analysis) remain valid — only the absolute PPL numbers are affected. > > **Status:** An architectural fix (`prefix_causal_registers`) has been implemented and verified to produce **exact 0.0** future-perturbation sensitivity in float64. > Re-training with the fixed architecture is required to obtain honest PPL numbers. This model card will be updated with corrected results once the re-run is complete. > > For full details, see the companion audit: [`Fock-PARFLM_Causal_Leak_Audit_Results.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Fock-PARFLM_Causal_Leak_Audit_Results.md). # [SUPERSEDED] Fock-PARFLM v2.1 with Depth-Conditioned Multi-Context Gaussian V_theta (OpenWebText, 27.23 PPL — under causal-leak review) The **OpenWebText-scale flagship** of the [Fock-PARFLM](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) conservative-language-model line. It replaces the MLP scalar potential Vθ with a **depth-conditioned, multi-context bank of Gaussian wells** (5 temporal-context heads × 8 wells = **40 attractors**), keeps the sparse pairwise potential Vφ and the full Fock register mechanism (32 registers, LIFO stack discipline, **reverse channel**), and trains it on **OpenWebText** to a validation perplexity of **27.23** using only **~2B training tokens**. > **Note:** The 27.23 PPL was measured with the original (non-prefix-causal) architecture and is subject to the causal leak described above. See the [disclaimer](#causal-leak-investigation-in-progress--reported-ppl-is-under-review) for details. This is a **from-scratch, attention-free** language model: every token's hidden state is evolved by a damped second-order dynamical system whose force is minus the gradient of a learned scalar potential, plus a sparse pairwise force, plus Fock creation/destruction/exchange forces. There is no softmax attention anywhere in the model. - **53.4M parameters** (GPT-2-Small class), **d = 384**, **L = 16** integration steps. - **27.23 PPL** on OpenWebText after **100K + 150K steps** (graduated 1B → 2B tokens). ⚠️ *Under causal-leak review.* - Two design decisions were decisive for reaching this number: the **Fock reverse channel** (removing it plateaus the model above 125 PPL — see the ablation below) and **untied input/output embeddings** (tied embeddings were unstable at this depth). This model is from the [Semantic Simulation](https://doi.org/10.5281/zenodo.19712427) framework. ## Table of Contents - [When to Use This Model](#when-to-use-this-model) - [Architecture](#architecture) - [What Made 27 PPL: Settings and Optimizations](#what-made-27-ppl-settings-and-optimizations) - [The Reverse-Channel Ablation](#the-reverse-channel-ablation) - [Geodesic Preservation: Riemannian Structure in the Dynamics](#geodesic-preservation-riemannian-structure-in-the-dynamics) - [How to Get Started](#how-to-get-started) - [Reproducing the Result](#reproducing-the-result) - [Training Details](#training-details) - [Evaluation Results](#evaluation-results) - [Available Artifacts](#available-artifacts) - [SPLM Family Overview](#splm-family-overview) - [Bias, Risks, and Limitations](#bias-risks-and-limitations) - [Citation](#citation) ## When to Use This Model This is a **research checkpoint** demonstrating that a conservative, attention-free, physics-informed language model can reach GPT-2-Small-class perplexity on a general web corpus. Choose it when you want to: - study a **non-transformer** sequence model with an explicit, interpretable force decomposition (self-energy Vθ, pair energy Vφ, Fock registers); - read **explicit semantic attractors** (40 Gaussian well centres, conditioned per temporal horizon and per layer) directly from the parameters; - experiment with **constant-memory inference** dynamics (the state is integrated step-by-step; there is no growing KV cache from a softmax attention). It is **not** a general-purpose chat or instruction model (see [Limitations](#bias-risks-and-limitations)). > ⚠️ **Causal-leak caveat:** Due to the identified causal leak in the reverse channel, this checkpoint's perplexity is inflated (optimistically low). Use the architecture and code for research purposes, but do not rely on the reported 27.23 PPL as a validated benchmark. > ⚠️ **Superseded caveat:** For new work at this scale, prefer the anisotropic Gaussian \\(V_\theta\\) + Fock-reg line linked in the notice at the top of this card — it has decisively outperformed this isotropic variant at every scale tested (TinyStories and, via the gamma-sweep diagnostics, OpenWebText). This card is kept for historical reference and link continuity only. ## Architecture ``` Input tokens x_1, ..., x_T | Untied token embedding E[x] + learned positional P[t] | For each of L=16 damped-Euler integration steps (shared force field): | +-- K=5 causal-EMA context channels: | xi^(m)_t = causal_ema(h, alpha_m) [horizons ~2 .. ~200 tokens] | +-- Depth-conditioned multi-context V_theta (Gaussian): | xi_g^(m) = xi^(m) + e_g^(m) [per-layer depth code e_g] | V_theta = sum_m Gaussian_well_bank_m(xi_g^(m), h) [5 heads x 8 wells] | f_theta = -grad_h V_theta [conservative self-force] | +-- Sparse pairwise V_phi (structural-competitive, 4 heads): | top-k=16 past tokens per query (Gumbel routing) | f_phi = -grad_h V_phi [autograd, sparse] | +-- Fock register pool (v2.1): | M=32 virtual registers, Q/K/V creation gates, d_k=64 | LIFO stack discipline, per-register tau/keys, ortho init | register repulsion (Gram penalty, lambda=0.05) | reverse channel (per-layer, stabilised, warmup 4000) [non-conservative] | *** CAUSAL LEAK SOURCE: register state is shared across all positions *** | f_fock = creation + destruction + exchange forces | +-- Total force: f = f_theta + f_phi + f_fock +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v +-- LayerNorm(h) | Logits = h @ W_out^T + b_out [UNTIED W_out; unigram-init bias b_out] ``` The mass *m* is a frozen per-token log-frequency (surprisal) lookup; the damping γ is a learned scalar fixed at 0.30 for this run. Note that the **effective damping is much smaller** than the nominal γ: the LayerNorm applied after each integration step rescales the hidden state, absorbing most of the velocity decay, so the dynamics are **heavily underdamped** even at γ = 0.30. The per-step force is minus the gradient of a scalar potential (plus the sparse pair force and the Fock forces), so each step is **conservative-by-construction** except for the single explicitly non-conservative reverse channel. | Parameter | Value | |---|---| | Hidden dim (d) | 384 | | Integration steps (L) | 16 | | Max sequence length | 1024 (trained at block 512) | | Vocab (GPT-2 BPE) | 50257 | | V_theta kind | depth-conditioned multi-context Gaussian | | V_theta heads × wells | 5 × 8 = 40 attractors | | Depth codes | per-layer (L × n_ctx × d), init std 0.02 | | Xi channels (K) | 5 | | Xi alpha inits | 0.50, 0.75, 0.95, 0.99, 0.995 | | V_phi kind | structural-competitive | | V_phi heads | 4 | | V_phi d_type / d_angle | 32 / 16 | | Top-k (sparse routing) | 16 | | Fock version | v2.1 | | Registers (M) | 32 | | Register d_k | 64 | | Stack discipline | LIFO | | Per-register tau / keys | Yes | | Reverse channel | Yes (per-layer, stabilised) — **causal leak source** | | Register repulsion | Gram, lambda 0.05 | | Mass model | logfreq (frozen surprisal) | | Damping gamma | 0.30 nominal, ~0.016 effective (LayerNorm prevents compounding; see [geodesic analysis](#interpretation)) | | Embeddings | UNTIED (separate W_out) + unigram bias | | lambda_V (V_theta regulariser) | 0.01 | | Prefix-causal registers | **No** (leak present; fix available but requires re-training) | | Total parameters | **53,378,075** | ## What Made 27 PPL: Settings and Optimizations The self-contained notebooks in this repo (they pull the model code from this repo's `code/` folder) reproduce the number. The settings that moved the needle, briefly: 1. **Depth-conditioned multi-context V_theta.** A single shared Gaussian well bank is reused across all L layers, but each layer adds a small learned depth code to the context before the bank projections. This gives each integration step a distinct effective potential at a cost of only ~L·n_ctx·d ≈ 31k parameters, instead of ~178M for fully untying the bank. Each of the 5 xi channels gets its own bank so attractors are conditioned on a specific temporal horizon rather than a blurred concatenation. 2. **Fock reverse channel (stabilised, per-layer).** The one non-conservative force. It is the single most important component for perplexity (see the ablation below), but it was also the dominant gradient-spike source. It is bounded with QK-norm, a soft-floored output RMS-norm, a 4000-step gate warmup, and a **per-layer** gate (rather than one global scalar) so its gradient does not aggregate into a high-variance knob. ⚠️ *This component is also the sole carrier of the [causal leak](#causal-leak-investigation-in-progress--reported-ppl-is-under-review): the reverse channel blends token content into a shared register state that is read by all positions, allowing future information to leak backward.* 3. **Untied embeddings + unigram output bias.** Tied embeddings were unstable at d=384 with this depth; untying the output head and initialising its bias from the log-unigram frequency fixed the long tail and stabilised training. 4. **WSD (Warmup–Stable–Decay) learning-rate schedule.** Peak LR is held through the middle of training so the wells have time to separate and specialise before the final cosine anneal. The 27.23 result comes from a **graduated two-phase** run: phase 1 (100K steps, 1B tokens) reaches ~63.7 PPL; a phase-2 continuation (150K steps, 2B tokens) with a fresh WSD cycle drives it to 27.23. 5. **Wider routing.** top-k doubled to 16, V_phi type/angle subspaces doubled to 32/16, and 4 V_phi heads — cheap in parameters, larger directional-force diversity. 6. **Per-group gradient clipping + watchdog.** Each top-level module is clipped to its own max-norm (V_phi and Fock gates to 0.3/0.1), with an EMA watchdog that reloads the last good checkpoint on sustained grad-norm blow-ups. This tames the second-order (`create_graph=True`) gradient cascade that appears late in training. 7. **Register repulsion.** A Gram-matrix penalty (lambda 0.05) keeps the 32 registers from collapsing onto a few effective directions. ## The Reverse-Channel Ablation The reverse channel is the sole non-conservative force in the model. Training the **identical architecture** with the reverse channel disabled (arm `e5a`) never breaks 120 PPL and plateaus around **125.94 PPL**; with the stabilised reverse channel on (arm `e5c`, this model) it reaches **27.23 PPL**. | Arm | Reverse channel | Best val PPL | Outcome | |---|---|---|---| | e5c (this model) | on (stabilised, per-layer) | **27.23** | continues improving through 250K steps | | e5a | off | 125.94 | plateaus early | The gap is not marginal — it is the difference between a working model and a stuck one. A purely conservative force field (all forces equal to minus a gradient of a potential) is measurably insufficient here; a bounded amount of directed, non-conservative information routing is what lets the dynamics match what attention provides. This is discussed at length in the companion note [`Fock_Mechanism_Ablation_Study_d384_OpenWebText.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Fock_Mechanism_Ablation_Study_d384_OpenWebText.md). > ⚠️ **Causal-leak note:** The reverse channel's dramatic impact on PPL (125.94 → 27.23) is now understood to be partly — and possibly largely — attributable to the causal leak it introduces. The reverse channel allows each token to read a register state that summarises *all* tokens in the window, including future ones. This is architecturally equivalent to giving the model access to bidirectional context for its predictions, which trivially lowers cross-entropy. The ablation remains valid as a *relative* comparison (arm e5a has no leak because it has no reverse channel), but the absolute PPL numbers in both arms are affected: arm e5c's 27.23 is leak-inflated (optimistically low), while arm e5a's 125.94 is a legitimate (leak-free) autoregressive perplexity. The true gap between "reverse channel on" and "reverse channel off" will only be known after re-training with `prefix_causal_registers=True`. ## Geodesic Preservation: Riemannian Structure in the Dynamics A defining claim of the Semantic Simulation framework is that the hidden-state trajectories are not arbitrary learned curves but approximate **Riemannian geodesics** on a manifold whose metric is induced by the learned potentials. We test this by computing the **damped geodesic residual** across a sweep of the damping coefficient. ### The experiment For each candidate damping value, we train a fresh d=384 L=16 model for 3,000 steps, then measure the per-layer geodesic residual $$R_\ell = \frac{\lVert h_{\ell+1} - h_\ell - \Delta t \cdot v_\ell \rVert}{\lVert h_{\ell+1} - h_\ell \rVert + \epsilon}$$ where the numerator is the deviation of the actual layer-step from the velocity-predicted geodesic step. Averaging across layers and validation batches gives the mean residual R̄(γ). If R̄ = 1 the dynamics are perfectly metric-compatible; values above 1 indicate the force field pushes the trajectory away from the geodesic. ### Results The figure below overlays validation perplexity (blue, left axis) and the damped geodesic residual R̄ (red, right axis) across eight gamma values. Two key findings emerge: ![PPL vs Geodesic Residual overlay (d=384, L=16)](geodesic/geodesic_overlay_d384.png) 1. **Near-geodesic at low damping.** At the geodesic-optimal γ = 0.05, the residual is R̄ = 1.041 — the dynamics deviate from the metric-compatible geodesic by only **4% per layer on average**. This confirms that the learned force field produces trajectories that are close to geodesics on the manifold it defines. 2. **PPL-geodesic coincidence breaks down at d=384.** The PPL-optimal γ is **0.25** (342 PPL), while the geodesic-optimal γ is **0.05** (R̄ = 1.041). At larger hidden dimensions (d=768, d=1024), the PPL and geodesic minima **coincide** at γ = 0.05. The d=384 breakdown is a **dimension-dependent phase transition**: smaller models prefer stronger damping for perplexity, even though it distorts the geodesic structure. | γ | γeff | Val PPL | Geodesic residual (R̄) | γgeo | Notes | |---|---|---|---|---|---| | 0.05 | 0.003 | 484 | **1.041** | 0.917 | geodesic-optimal | | 0.10 | 0.006 | 419 | 1.050 | 0.880 | | | 0.15 | 0.009 | 350 | 1.309 | 0.974 | | | 0.20 | 0.011 | 397 | 1.345 | 0.871 | | | **0.25** | **0.014** | **342** | 1.418 | 0.988 | **PPL-optimal** | | 0.30 | 0.016 | 354 | 2.115 | 0.995 | this model's training gamma | | 0.40 | 0.021 | 741 | 1.636 | 0.942 | | | 0.50 | 0.026 | 370 | 1.758 | 0.910 | | γgeo is the **recovered intrinsic damping** — the γ value that would minimise R̄ if the residual were re-evaluated against this specific trajectory rather than the nominal training γ. It clusters tightly at 0.87–1.00 across the entire sweep, essentially independent of the nominal training γ. The per-layer heatmap reveals that geodesic violation concentrates in the **middle layers** (layers 6-10) at high damping, while early and late layers stay near-geodesic — consistent with the middle layers carrying the heaviest semantic-routing burden. **Note on γ=0.30 (this model's training gamma).** The R̄=2.115 at γ=0.30 is the highest in the sweep — a non-monotonic spike above both neighbors (γ=0.25: 1.418, γ=0.40: 1.636). This does **not** mean the model has left the Riemannian manifold or stopped following geodesics. The residual tests compliance with the damped geodesic equation *at the nominal γ*; a large R̄ means the nominal hyperparameter is a poor match for the trajectory's intrinsic damping, not that the trajectory is non-geodesic. The γgeo column recovers the damping the trajectory actually exhibits: for this checkpoint γgeo ≈ 0.995 (near-critically-damped). If R̄ were re-evaluated at γeval = γgeo, it would drop back toward R̄ ≈ 1. The model follows a damped geodesic *at its own intrinsic damping* regardless of the training hyperparameter — and that is all the geometric toolkit needs. Both the energy-dissipation hallucination detector and native chain-of-thought via Fock register dynamics require only a valid Jacobi metric (guaranteed by the exact potential Vθ) and compliance with *some* damped geodesic equation, not that the nominal γ matches the intrinsic one. ### Interpretation The fact that R̄ ≈ 1 at low damping is not a trivial outcome. The model was trained to minimise cross-entropy, not to produce geodesic trajectories. That the learned force field happens to be nearly metric-compatible is an emergent structural property — the scalar and pairwise potentials organise the hidden-state manifold so that the minimum-energy paths are also the most predictive paths. This is the central prediction of the Lagrangian framework: conservative forces should produce geodesic-like dynamics because the Euler-Lagrange equations of a conservative system are the geodesic equations of the kinetic-energy metric. An important caveat when interpreting the nominal γ values: the LayerNorm after each integration step rescales the hidden state, preventing the per-step velocity decay from compounding across layers. Without LayerNorm, the velocity after L = 16 steps would be reduced to v0 / (1 + γ)L — at γ = 0.30, this means only **1.5%** of the initial velocity survives, a catastrophically overdamped regime. With LayerNorm resetting the state scale each step, the decay does not compound, and the total velocity reduction over L layers is closer to a single-step reduction. The **effective damping** γeff (the γeff column in the table above) is estimated as the decompounded per-step equivalent: $$\gamma_{\text{eff}} = (1 + \gamma)^{1/L} - 1 \approx \gamma / L$$ At γ = 0.30 with L = 16, this gives γeff ≈ 0.016 — an **18× reduction** from the nominal value. Even the PPL-optimal γ = 0.25 corresponds to γeff ≈ 0.014. The model operates in a **heavily underdamped regime** across the entire sweep range, which explains why d=384 can tolerate nominally high γ values without over-dissipating. At larger hidden dimensions (d=768, d=1024), the LayerNorm correction is relatively weaker, and the nominal and effective damping converge, which is consistent with the PPL-geodesic coincidence reappearing at γ = 0.05. The geodesic analysis data and images are in the `geodesic/` folder. For the full cross-scale analysis (d=384, d=768, d=1024), see the companion note [`Geodesic_Preservation_Experiment.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Geodesic_Preservation_Experiment.md). ## How to Get Started The full model-code closure is bundled in this repo under `code/`, so no external clone is needed. ```python import sys, torch from pathlib import Path from huggingface_hub import snapshot_download, hf_hub_download REPO = "dimitarpg13/semsimula-fock-parflm-depthcond-vtheta-openwebtext" BEST = "checkpoints/fock_dcvt_owt_xi5long_topk16_dt32da16_mh4_dcvt5x8_ob_untied_wsd_e5c_plgate_rep0.05_best.pt" # 1. Download the bundled model code and put it on sys.path code = Path(snapshot_download(repo_id=REPO, allow_patterns=["code/**"])) / "code" for sub in ["", "parf", "multixi", "sarf_mass_variant", "energetic_minima"]: sys.path.insert(0, str(code / sub) if sub else str(code)) from model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig from model_gaussian_vtheta import DepthConditionedMultiContextGaussianVTheta, install_depth_routing # 2. Load the best checkpoint (it carries the exact model_cfg it was built with) state = torch.load(hf_hub_download(repo_id=REPO, filename=BEST), map_location="cpu", weights_only=False) # 3. Point the mass model at the hosted surprisal table, then rebuild the config. # (The model reads logfreq_path at construction; the saved path is a Colab path.) mc = dict(state["model_cfg"]) mc["logfreq_path"] = hf_hub_download(repo_id=REPO, filename="data/logfreq_surprisal_openwebtext.npy") cfg = FockMultiXiPARFConfig(**mc) model = FockMultiXiPARFLM(cfg) # 4. Swap in the depth-conditioned multi-context Gaussian V_theta and load weights. model.V_theta = DepthConditionedMultiContextGaussianVTheta( d=cfg.d, K=8, n_ctx=5, n_layers=cfg.L, ) install_depth_routing(model) model.load_state_dict(state["model_state_dict"], strict=False) model.eval() print(f"Parameters: {model.num_params():,}") # 53,378,075 print(f"Checkpoint step {state['step']:,} val PPL {state['val_ppl']:.2f}") # step 150,000 PPL 27.23 ``` The exact V_theta construction (well count, precision init, depth-code std) matches the training notebooks; the simplest way to rebuild the model identically is to run the model-config and V_theta-swap cells of `notebooks/colab_fock_depthcond_vtheta_openwebtext.ipynb`, which construct it from the same `model_cfg` fields and then load the checkpoint. ## Reproducing the Result Two self-contained Colab notebooks are included. Both pull all model code from **this** HF repo (no GitHub clone) and download OpenWebText from the Hub. | Notebook | Role | |---|---| | `notebooks/colab_fock_depthcond_vtheta_openwebtext.ipynb` | Phase 1 — 100K steps, 1B tokens, WSD, reaches ~63.7 PPL | | `notebooks/colab_fock_depthcond_vtheta_openwebtext_ext.ipynb` | Phase 2 — 150K-step continuation, 2B tokens, fresh WSD cycle, reaches 27.23 PPL | Run phase 1 first, then phase 2. The extension notebook will **automatically seed** from the hosted phase-1 checkpoint (`checkpoints/..._step99000_best.pt`) if no local/Drive checkpoint is present, so phase 2 can also be run on its own. > ⚠️ **Note:** These notebooks use the original (leaky) architecture. To reproduce with the causal-leak fix, set `prefix_causal_registers=True` in the `FockMultiXiPARFConfig`. See the [companion audit](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Fock-PARFLM_Causal_Leak_Audit_Results.md) for the updated training notebooks. ## Training Details ### Training Data [OpenWebText](https://huggingface.co/datasets/Skylion007/openwebtext) — an open reproduction of the WebText corpus, tokenized with GPT-2 BPE (vocab 50257). Training consumed a **graduated token budget** (1B tokens in phase 1, 2B tokens in phase 2; ~2.05B tokens total). **Validation set (held-out, no data leakage).** When OpenWebText is first streamed and tokenised, the code requests `MAX_TRAIN_TOKENS + 2,000,000` total tokens. The **last 2M tokens** of the stream are sliced off as the validation set; the **first N tokens** become the training set. The two are cached as separate files (`openwebtext_val_2M.npy` and `openwebtext_train_{N}M.npy`) with zero token-level overlap. When the token budget graduated from 1B to 2B, the training pool grew but the validation set remained the **same 2M-token held-out slice**, so all PPL numbers across phases are directly comparable. Each evaluation step draws 5 random batches of length 512 from the validation set, computes mean cross-entropy loss, and exponentiates: PPL = exp(L̄val). The validation file is included in this repo at `data/openwebtext_val_2M.npy`. ### Training Procedure | Hyperparameter | Value | |---|---| | Optimizer | AdamW | | Peak learning rate | 1.5e-4 (phase 2), 3e-4 (phase 1) | | LR schedule | WSD (warmup–stable–decay), floor 1.5e-5 | | Warmup | 2,000 steps | | Weight decay | 0.01 | | Effective batch | 16 (batch 8 × grad-accum 2) | | Block size | 512 | | Global grad clip | 1.0 (per-group; V_phi 0.3, Fock gates 0.1) | | lambda_V (V_theta regulariser) | 0.01 | | Register repulsion coeff | 0.05 | | Steps | 100,000 (phase 1) + 150,000 (phase 2) | | Seed | 0 | | Hardware | 1× NVIDIA H100 80GB (Google Colab) | Memory-critical optimizations enabled during training: gradient (layer) checkpointing over the integration steps, a gathered sparse-Vφ path, and LayerNorm-before-distance in the routing scorer. ## Evaluation Results ### OpenWebText Validation Perplexity | Model | PPL | Params | Reverse channel | Causal leak | |---|---|---|---|---| | **Fock-PARFLM v2.1 (depth-cond multi-context Gaussian, this model)** | **27.23** ⚠️ | 53.4M | on | **Yes — under review** | | Fock-PARFLM v2.1, reverse channel off (ablation arm e5a) | 125.94 | 53.4M | off | No (no reverse channel) | > ⚠️ The 27.23 PPL is measured with the original architecture which has a confirmed causal leak via the reverse channel's shared register state. The honest (leak-free) PPL of this checkpoint is expected to be substantially higher. The ablation arm e5a (reverse channel off) is leak-free and its 125.94 PPL is a valid autoregressive perplexity. See the [causal-leak disclaimer](#causal-leak-investigation-in-progress--reported-ppl-is-under-review) at the top of this card. PPL is evaluated on a **held-out 2M-token validation slice** of OpenWebText (see [Training Data](#training-data) for the split protocol). There is no overlap between training and validation tokens. PPL here is on OpenWebText, a general web corpus. It is **not** comparable to the TinyStories perplexities reported for the smaller SPLM-family cards (TinyStories is a far easier, synthetic corpus). ### Best checkpoints over training | Checkpoint | Step | Val PPL | Causal leak status | |---|---|---|---| | `..._best.pt` (= step 150000) | 150,000 (phase 2) | **27.23** ⚠️ | Leak present | | `..._step140500_best.pt` | 140,500 | 27.90 ⚠️ | Leak present | | `..._step135000_best.pt` | 135,000 | 29.00 ⚠️ | Leak present | | `..._step99000_best.pt` | 99,000 (phase-1 end) | 63.69 ⚠️ | Leak present | ### Learned Xi-Channel Decay Rates The final learned alpha values are `[0.172, 0.322, 0.498, 0.639, 0.965]` (init `[0.50, 0.75, 0.95, 0.99, 0.995]`), spreading the five causal-EMA channels across short-to-long temporal horizons. ## Available Artifacts | Path | Description | |---|---| | `checkpoints/..._best.pt` | Best checkpoint (27.23 PPL ⚠️, step 150000) — identical to step150000 | | `checkpoints/..._step140500_best.pt` | 27.90 PPL milestone | | `checkpoints/..._step135000_best.pt` | 29.00 PPL milestone | | `checkpoints/..._step99000_best.pt` | End of phase 1 (63.69 PPL); seeds the extension notebook | | `code/` | Full self-contained model-code closure (parf, multixi, sarf_mass_variant, energetic_minima, data_module) | | `notebooks/colab_fock_depthcond_vtheta_openwebtext.ipynb` | Phase-1 training notebook | | `notebooks/colab_fock_depthcond_vtheta_openwebtext_ext.ipynb` | Phase-2 extension notebook | | `data/openwebtext_val_2M.npy` | Held-out validation tokens | | `data/logfreq_surprisal_openwebtext.npy` | Frozen per-token surprisal (mass model) | | `results/training_log_phase1.jsonl` | Phase-1 per-step metrics | | `results/training_log_phase1_phase2.jsonl` | Full 250K-step per-step metrics | | `geodesic/geodesic_overlay_d384.png` | PPL vs geodesic residual overlay plot | | `geodesic/geodesic_per_layer_d384.png` | Per-layer geodesic residual heatmap | | `geodesic/gamma_geo_recovery_d384.png` | Recovered intrinsic damping across gamma | | `geodesic/geodesic_results.json` | Full per-gamma, per-layer geodesic residual data | | `config.json` | Model + training configuration and headline results | Each checkpoint is a dict with `model_state_dict`, `optimizer_state_dict`, `model_cfg`, `train_cfg`, `step`, `val_ppl`, `val_loss`, and the learned `xi_alphas`. ## SPLM Family Overview This model is (was) part of the **Semantic Simulation SPLM family**; it has been **removed from the collection's front page** as superseded (see the notice at the top of this card) but the table below is left as it was at publication time, for historical reference. The other cards are smaller TinyStories models; this was the first OpenWebText-scale member. | Model | Design | Corpus | HuggingFace | |---|---|---|---| | Multi-Xi SPLM | Pure scalar potential | TinyStories | [semsimula-splm-multixi](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) | | Multi-Xi PARFLM | Scalar + pairwise | TinyStories | [semsimula-parflm-multixi](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi) | | Fock-PARFLM v2.1 (MLP) | + Fock registers | TinyStories | [semsimula-fock-parflm](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) | | Fock-PARFLM v2.1 (SQ3) | + structured V_theta | TinyStories | [semsimula-fock-parflm-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-structured-vtheta) | | Fock-Attention | Fock + attention | TinyStories | [semsimula-fock-attention](https://huggingface.co/dimitarpg13/semsimula-fock-attention) | | Hybrid SPLM+Attn | Attention + SPLM refinement | TinyStories | [semsimula-hybrid-splm](https://huggingface.co/dimitarpg13/semsimula-hybrid-splm) | | **[SUPERSEDED] Fock-PARFLM v2.1 (depth-cond multi-context Gaussian)** | **+ depth-conditioned Gaussian V_theta** | **OpenWebText** | **this model** | **For the current OpenWebText-scale line (anisotropic Gaussian \\(V_\theta\\) + Fock-reg), see the superseded notice at the very top of this card.** **Collection:** [Semantic Simulation SPLM Model Family](https://huggingface.co/collections/dimitarpg13/semantic-simulation-splm-model-family) ## Bias, Risks, and Limitations - **Research checkpoint only.** A proof-of-concept that a conservative, attention-free dynamical language model can reach GPT-2-Small-class perplexity on a general corpus — not a production system. - **OpenWebText only, English only.** No instruction tuning, no RLHF/DPO, no safety filtering. Generations reflect the biases of web text. - **Small scale.** 53.4M parameters, d=384. - **Training cost of the force field.** The self-force fθ = -∇hVθ and the Fock forces are computed with a second-order graph (`create_graph=True`); this is the source of the late-training gradient spikes the watchdog and per-group clipping are designed to catch. See the companion note [`Training_Instabilities_in_Fock-PARFLM_with_structured_V_theta.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Training_Instabilities_in_Fock-PARFLM_with_structured_V_theta.md). - **Reverse channel is load-bearing.** The single non-conservative force cannot be removed without losing most of the model's quality (125.94 vs 27.23 PPL). - ⚠️ **Confirmed causal leak in the reverse channel.** The Fock reverse channel introduces a causal leak by blending each token's content into a global register state shared across all sequence positions within the same integration step. Future-token information flows backward into past-token predictions, violating autoregressive causality. A post-publication causal probe on a later checkpoint from the same architecture measured +3.51 nats of NLL inflation (~33× PPL factor). An architectural fix (`prefix_causal_registers`) has been implemented and verified, but this checkpoint was trained without it. **The reported 27.23 PPL is not a valid autoregressive perplexity** — re-training with the fix is required to obtain honest numbers. See the [full audit](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Fock-PARFLM_Causal_Leak_Audit_Results.md). ## Citation ```bibtex @misc{Gueorguiev2026SemSim, author = {Gueorguiev, Dimitar P.}, title = {Semantic Simulation: A Prescriptive Lagrangian Framework for Efficient Semantic Inference --- A Conservative-by- Construction Language Model and the Shared-Potential Separator, with a Correspondence to Joint Embedding Predictive Architectures}, year = {2026}, publisher = {Zenodo}, doi = {10.5281/zenodo.19712427}, url = {https://doi.org/10.5281/zenodo.19712427}, note = {Companion code repository: \url{https://github.com/dimitarpg13/semsimula-paper}} } ``` ## Environmental Impact - **Hardware:** 1× NVIDIA H100 80GB (Google Colab) - **Training:** 250,000 steps total (100K + 150K), ~2.05B tokens - **Carbon footprint:** small; a single-GPU research run, estimated on the order of a few kg CO2