dimitarpg13/semsimula-fock-parflm-depthcond-vtheta

Fock-PARFLM v2.1 with Depth-Conditioned Multi-Context Gaussian V_theta (TinyStories)

The TinyStories-scale member of the depth-conditioned multi-context Gaussian $V_{\theta }V\_\backslash theta$ family, whose flagship is the OpenWebText-scale Fock-PARFLM (27.23 PPL, 53.4M params). This checkpoint uses the same $V_{\theta }V\_\backslash theta$ design — a bounded mixture of Gaussian wells, one bank per xi-context channel, with a small learned per-layer depth code — at the smaller Fock-PARFLM v2.1 TinyStories scale (d=256, L=8), alongside the same MLP - $V_{\theta }V\_\backslash theta$ and SQ3-structured siblings.

Unlike its two Fock-PARFLM siblings, this checkpoint needed no causal-leak re-training: it was trained from step 0 with prefix_causal_registers=True and its bit-exact future-perturbation probes and honest-vs-standard PPL checks pass cleanly throughout training. See Training Details for the probe evidence.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Model
• Architecture
• The Analytical Form of the Potential
• How to Get Started
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

When to Use This Model

Choose this variant over the MLP-based or SQ3-structured Fock-PARFLM v2.1 when a structurally bounded, per-horizon-conditioned potential is what you want to study — not when you want the best TinyStories PPL at this scale:

Priority	Depth-cond. multi-context Gaussian (this model)	SQ3 structured (sibling)	MLP (baseline)
Boundedness	V_theta in [-4, 0] by construction, force magnitude capped	Unbounded (log-sum-exp mixture)	Unbounded
Interpretability	32 explicit attractor centres, one bank per temporal horizon	8 explicit attractor centres	Black-box
Per-layer specialisation	Yes — shared bank + learned per-layer depth-code shift	No (one shared bank, no depth code)	No
Raw PPL	16.33	10.90	9.70
Causal-leak history	Trained leak-free natively (no re-training needed)	Re-trained with the fix (+0.54 PPL tax)	Re-trained with the fix (+0.40 PPL tax)

Bottom line: at this small scale (d=256, L=8, 16k steps), the boundedness of the Gaussian mixture costs a large amount of expressivity relative to the unbounded SQ3 and MLP potentials — a 6.63 PPL gap to the honest MLP baseline. The same potential family reaches a competitive 27.23 PPL on OpenWebText, but only once combined with a much larger scale (d=384, L=16), five xi-context channels instead of four, wider sparse routing, register repulsion, untied embeddings, and a graduated WSD schedule. This TinyStories checkpoint isolates the potential design in the small-scale regime and shows it is not, by itself, sufficient to match the unbounded alternatives there — see Evaluation Results.

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps (layer index g = 0..7):
       |
       +-- K-EMA channels: xi^(m)_t = causal_ema(h, alpha_m)   [n_ctx=4 channels]
       |
       +-- Depth-conditioned multi-context V_theta (Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     V_m = -sum_k w_k^m exp(-0.5 a_k^m (h - mu_k^m)^2)  [8 wells per context]
       |     V_theta = sum_m V_m(xi_g^(m), h)               [4 contexts, 32 wells total]
       |     f_theta = -analytical_grad_h V_theta            [closed-form, bounded]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=16 virtual registers with Q/K/V creation gates
       |     LIFO stack discipline, salience decay
       |     Per-register tau and key subspaces
       |     Reverse channel (non-conservative exchange), prefix-causal (leak-free)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]

Parameter	Value
Hidden dim (d)	256
Layers (L)	8
V_theta kind	Depth-conditioned multi-context Gaussian (bounded mixture)
V_theta contexts (n_ctx)	4 (one bank per xi channel)
Wells per context (K)	8
Total attractors	32
Depth codes	per-layer additive shift, shape (L=8, n_ctx=4, d=256), init std 0.02
Xi channels (K_xi)	4
V_phi kind	structural_competitive
V_phi hidden	128
Top-k (sparse routing)	8
Gumbel tau	1.0 (init), 0.3 (min)
Fock version	v2.1
Registers (M)	16
Register d_k	64
Stack discipline	LIFO
Reverse channel	Yes — prefix-causal, leak-free
Per-register tau/keys	Yes
Gathered V_phi	Yes
Per-layer V_phi scale	Yes
LN before distance	Yes
Layer checkpoint	Yes
Mass model	logfreq (frozen surprisal lookup)
Damping gamma	0.30 nominal (LayerNorm prevents compounding; effective damping is much smaller — see the MLP sibling's note)
lambda_V (V_theta regularisation)	0.01
Prefix-causal registers	Yes — trained natively with the fix, not retrofitted
Total parameters	18,213,899
V_theta parameters	4,227,104

The Analytical Form of the Potential

Each of the 4 xi-context channels gets its own bank of 8 diagonal-precision Gaussian wells. For context $mm$, well $kk$, the bank contributes

$$V_m({\xi }^{(m)},h)=-\sum _{k=1}^8{w}_k^{(m)}({\xi }^{(m)})\exp (-\frac{1}{2}\sum _{d=1}^{256}{a}_{k,d}^{(m)}({\xi }^{(m)}){(h_d-{\mu }_{k,d}^{(m)}({\xi }^{(m)}))}^2)V\_m(\backslash xi^\{(m)\},\; h)\; =\; -\backslash sum\_\{k=1\}^\{8\}\; w\_k^\{(m)\}(\backslash xi^\{(m)\})\; \backslash exp\backslash left(-\backslash frac\{1\}\{2\}\; \backslash sum\_\{d=1\}^\{256\}\; a\_\{k,d\}^\{(m)\}(\backslash xi^\{(m)\})\; \backslash left(h\_d\; -\; \backslash mu\_\{k,d\}^\{(m)\}(\backslash xi^\{(m)\})\backslash right)^2\backslash right)$$

where the mixture weights $w_k^{(m)}w\_k^\{(m)\}$ are a softmax over a linear projection of ${\xi }^{(m)}\backslash xi^\{(m)\}$ (so they sum to 1 per context), and the per-dimension precisions $a_{k,d}^{(m)}a\_\{k,d\}^\{(m)\}$ are strictly positive via a softplus projection. Because every term is a negative exponential bounded in $(-1,0](-1,\; 0]$, each bank's contribution is bounded in $[-1,0][-1,\; 0]$ — this is the structural fix that gives the model a provably finite potential and force, in contrast to the SQ3 sibling's unbounded log-sum-exp mixture.

Depth conditioning. Rather than training 8 independent well banks (one per integration layer, which would multiply the parameter count by $L=8L=8$), a single shared bank per context is reused across all layers. Each layer $gg$ instead learns a small additive depth code $e_g^{(m)}\in {\mathbb{R}}^{256}e\_g^\{(m)\}\; \backslash in\; \backslash mathbb\{R\}^\{256\}$ that shifts the context before the bank sees it:

$${\xi }_g^{(m)}={\xi }^{(m)}+e_g^{(m)},V_{\theta }(\xi ,h)=\sum _{m=1}^4{V}_m({\xi }_g^{(m)},h)\backslash xi\_g^\{(m)\}\; =\; \backslash xi^\{(m)\}\; +\; e\_g^\{(m)\},\; \backslash qquad\; V\_\backslash theta(\backslash xi,\; h)\; =\; \backslash sum\_\{m=1\}^\{4\}\; V\_m(\backslash xi\_g^\{(m)\},\; h)$$

Since $e_g^{(m)}e\_g^\{(m)\}$ is constant with respect to $hh$, the gradient is unaffected by this additive shift of the input, and the resulting total potential over all 4 contexts is bounded in $[-4,0][-4,\; 0]$.

Force (closed form):

$$f_{\theta }=-{\mathrm{\nabla }}_h{V}_{\theta }=\sum _{m=1}^4\sum _{k=1}^8{w}_k^{(m)}{a}_k^{(m)}\odot (h-{\mu }_k^{(m)})\exp (-\frac{1}{2}{a}_k^{(m)}\cdot {(h-{\mu }_k^{(m)})}^2)f\_\backslash theta\; =\; -\backslash nabla\_h\; V\_\backslash theta\; =\; \backslash sum\_\{m=1\}^\{4\}\; \backslash sum\_\{k=1\}^\{8\}\; w\_k^\{(m)\}\; a\_k^\{(m)\}\; \backslash odot\; \backslash left(h\; -\; \backslash mu\_k^\{(m)\}\backslash right)\; \backslash exp\backslash left(-\backslash frac\{1\}\{2\}\; a\_k^\{(m)\}\; \backslash cdot\; \backslash left(h\; -\; \backslash mu\_k^\{(m)\}\backslash right)^2\backslash right)$$

The Gaussian factor decays to zero away from each well centre, so the force magnitude is bounded (it peaks at a finite radius from each ${\mu }_k^{(m)}\backslash mu\_k^\{(m)\}$ and vanishes both at the centre and far away) — unlike SQ3's quadratic force, which grows without bound as $hh$ moves away from a well.

Empirically bounded, but far from the theoretical ceiling. The theoretical range of $V_{\theta }V\_\backslash theta$ is $[-4,0][-4,\; 0]$, but on this checkpoint the observed range over validation batches is only [-0.624, -0.00002] (mean -0.017, std 0.024) — the 32 wells are only weakly activated at this checkpoint. This is the flattest, most compressed landscape of any Fock-PARFLM $V_{\theta }V\_\backslash theta$ variant published so far (see Evaluation Results), and plausibly part of why the PPL gap to the unbounded siblings is large at this scale: the force budget carried by $V_{\theta }V\_\backslash theta$ here is tiny relative to $V_{\varphi }V\_\backslash phi$ and the Fock registers.

For the full derivation, boundedness proof, and the SARFGaussianVTheta frozen-anchor variant, see the companion note: Structured_VTheta_Design_and_Theory.md and the OpenWebText flagship's write-up of the same design at scale.

How to Get Started

import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_gaussian_vtheta import DepthConditionedMultiContextGaussianVTheta, install_depth_routing

# -- Build base model --
config = FockMultiXiPARFConfig(
    vocab_size=50257, d=256, L=8,
    v_hidden=1024, v_depth=3,
    max_len=1024,
    fixed_gamma=0.30, xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True, mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
    fock_version="v2",
    n_registers=16, d_k=64,
    tau_create_init=8.0,
    register_salience_decay=0.5,
    register_salience_threshold=0.01,
    stack_discipline=True,
    reverse_channel=True,
    per_register_tau=True,
    per_register_keys=True,
    ortho_register_init=True,
    prefix_causal_registers=True,
)
model = FockMultiXiPARFLM(config)

# -- Swap in the depth-conditioned multi-context Gaussian V_theta --
model.V_theta = DepthConditionedMultiContextGaussianVTheta(d=256, K=8, n_ctx=4, n_layers=8)
install_depth_routing(model)  # wires the per-layer index into V_theta's depth-code lookup

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-fock-parflm-depthcond-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly (32 wells: 4 contexts x 8 wells) --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                                # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)              # (1, 64, 4, 8, 256)
    print(f"Attractor centres shape: {centres.shape}")

Available Artifacts

File	Description
checkpoint/ckpt_best.pt	Best checkpoint (16.33 PPL at step 14,400), trained natively leak-free
training_log.jsonl	Per-step training metrics, including causal_probe and trained_leak_probe events
training_curve_gaussian.png	Training/validation loss curves
v_theta_hist_gaussian.png	V_theta output distribution histogram
landscape_stats_gaussian.json	V_theta landscape statistics (mean, std, min, max, range)
model_gaussian_vtheta.py	Gaussian V_theta classes (mixture, SARF-anchored, multi-context, depth-conditioned)
model_structured_vtheta.py	Base class (StructuredVThetaBase) required by model_gaussian_vtheta.py
config.json	Model configuration

Training Details

Training Data

TinyStories --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens.

Training Procedure

The base architecture is identical to Fock-PARFLM v2.1. The only modification is the $V_{\theta }V\_\backslash theta$ replacement: a single shared DepthConditionedMultiContextGaussianVTheta bank (4 contexts x 8 wells) plus per-layer depth codes are constructed in place of the 3-layer MLP, before training begins from scratch. The pairwise $V_{\varphi }V\_\backslash phi$ (competitive structural MLP, hidden=128, top-k=8) and Fock register pool (16 registers, LIFO, reverse channel) are unchanged, and prefix_causal_registers=True is set from the start.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	5e-4 (cosine decay)
Warmup steps	400
Weight decay	0.01
Gradient clipping	1.0
Batch size	16
Block size	512
Training steps	16,000
lambda_V (V_theta regularisation)	0.01
Hardware	NVIDIA A100 40GB (Google Colab)

Causal-Leak Verification (no fix needed — trained natively leak-free)

Because this run started with prefix_causal_registers=True from step 1 (rather than being re-trained after the fact like the MLP and SQ3 siblings), the training log carries the same probes as a matter of routine certification, not remediation:

step	causal_probe_max_delta	causal_probe_passed	honest vs. standard PPL diff (nats)
4,000	0.0	True	—
8,000	0.0	True	+0.0077 ± 0.0192 SE
12,000	0.0	True	—
16,000	0.0	True	+0.0247 ± 0.0164 SE

The deterministic architectural probe is bit-exact 0.0 at every checkpoint, and the honest-vs-standard PPL differences are both within ~1.5 standard errors of zero — not statistically distinguishable from no leak. No causal-leak caveat applies to the 16.33 PPL reported on this card.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_gaussian_vtheta.ipynb (companion repo) --- Colab notebook with the depth-conditioned multi-context Gaussian arm, GDrive output, checkpointing, and live progress display.

Evaluation Results

TinyStories Validation Perplexity

Model	PPL	Params	Analytical V_theta grad	Bounded V_theta	Causal leak
Matched Attention (baseline)	7.81	19.5M	---	---	No
Fock-PARFLM v2.1 (MLP)	9.70	17.4M	No	No	Fixed & verified
Fock-PARFLM v2.1 (SQ3)	10.90	18.2M	Yes	No	Fixed & verified
Fock-PARFLM v2.1 (depth-cond. Gaussian, this model)	16.33	18.2M	Yes	Yes	Never leaked

All PPL values in this table are honest, leak-free numbers. This model and the SQ3 sibling have nearly identical total parameter counts (both ~18.2M, dominated by their respective $V_{\theta }V\_\backslash theta$ hypernetworks) yet very different PPL — the gap here is attributable to the design trade-off of a bounded potential, not to parameter budget.

PPL is evaluated on the TinyStories validation split; see Training Data.

V_theta Landscape Statistics

Metric	This model (depth-cond. Gaussian)	Fock-PARFLM (SQ3)	Fock-PARFLM (MLP)
Mean V_theta	-0.017	0.017	n/a (MLP has no comparable scalar readout)
Std V_theta	0.024	0.42	n/a
Range	0.624	16.0	n/a
Theoretical bound	[-4, 0]	(-∞, +∞)	(-∞, +∞)

This is the flattest, most compressed $V_{\theta }V\_\backslash theta$ landscape of any Fock-PARFLM variant published so far — a direct, structural consequence of the bounded Gaussian-mixture design rather than something the optimiser found on its own.

Learned Xi-Channel Decay Rates

The final learned alpha values $[{\alpha }_1,\dots ,{\alpha }_4]=[0.321,0.544,0.854,0.908][\backslash alpha\_1,\; \backslash ldots,\; \backslash alpha\_4]\; =\; [0.321,\; 0.544,\; 0.854,\; 0.908]$ (initialised at $[0.25,0.5,0.75,0.95][0.25,\; 0.5,\; 0.75,\; 0.95]$) settle to a somewhat different spread than the MLP and SQ3 siblings' $\approx [0.11\text{–}0.14,0.55\text{–}0.59,0.79\text{–}0.84,0.95\text{–}0.97]\backslash approx\; [0.11\backslash text\{--\}0.14,\; 0.55\backslash text\{--\}0.59,\; 0.79\backslash text\{--\}0.84,\; 0.95\backslash text\{--\}0.97]$, suggesting the depth-conditioned Gaussian potential shifts slightly more weight onto the two longer-horizon xi channels.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Model	Design	PPL	HuggingFace
Multi-Xi SPLM (MLP)	Pure scalar potential	11.51	semsimula-splm-multixi
Multi-Xi PARFLM (MLP)	Scalar + pairwise forces	12.06	semsimula-parflm-multixi
Fock-PARFLM v2.1 (MLP)	PARFLM + Fock registers	9.70	semsimula-fock-parflm
Fock-PARFLM v2.1 (SQ3)	Structured + pairwise + Fock	10.90	semsimula-fock-parflm-structured-vtheta
Fock-PARFLM v2.1 (depth-cond. Gaussian)	Bounded multi-context + pairwise + Fock	16.33	this model
Fock-PARFLM v2.1 (depth-cond. Gaussian, OpenWebText scale)	Bounded multi-context + pairwise + Fock, d=384 L=16	27.23	semsimula-fock-parflm-depthcond-vtheta-openwebtext
Fock-Attention	Fock + attention	9.42	semsimula-fock-attention
Hybrid SPLM+Attn	Attention + SPLM refinement	8.50	semsimula-hybrid-splm

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Research checkpoint only. This model is a proof-of-concept for bounded, depth-conditioned scalar potentials in Fock-augmented architectures, not a production system.
• TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
• English only. No multilingual capability.
• Small scale. 18.2M parameters, 256-dim hidden states.
• No safety training. No RLHF, DPO, or safety filtering has been applied.
• V_phi and Fock forces still use autograd. Only the $V_{\theta }V\_\backslash theta$ gradient is analytical; the $V_{\varphi }V\_\backslash phi$ pairwise force and Fock register forces still require torch.autograd.grad.
• Large expressivity gap at this scale. The 6.63 PPL gap to the honest MLP baseline (16.33 vs 9.70) is substantially larger than the SQ3 sibling's 1.20 PPL gap. The bounded-potential design that lets the OpenWebText-scale flagship reach 27.23 PPL does not, by itself, transfer its advantage down to this much smaller model/training budget — see When to Use This Model.
• Weakly activated wells. The empirical $V_{\theta }V\_\backslash theta$ range (0.624) is far below the theoretical bound (4.0), indicating the 32 wells are only lightly used at this checkpoint; this may partly explain the gap to the unbounded siblings.
• No causal-leak issue. Unlike the MLP and SQ3 siblings, this checkpoint was trained natively with prefix_causal_registers=True and needed no re-training or PPL correction; see Causal-Leak Verification.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: NVIDIA A100 40GB (Google Colab)
• Training time: ~3 hours (16,000 steps)
• Carbon footprint: Estimated less than 2 kg CO2