You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This repo contains a research checkpoint for an Online Hebbian mlp-mixer language model. Access is granted manually for collaborators.
Log in or Sign Up to review the conditions and access this model content.
Online Hebbian v10 Growing 32->528 mE16
This repo contains the Lightning checkpoint for the mlp-mixer Online Hebbian v10 growing-memory language-model run used in the experiment notes as 32->128, 32->256, and 32->528.
Those labels are eval-time linear active-feature schedules applied to the same trained checkpoint, not three separately trained model files. The trained schedule starts with 32 active bilinear features and expands by 16 features every 64 tokens, reaching 528 features at L=2048.
Provenance
- W&B: https://wandb.ai/hazy-research/mlp-mixer/runs/m05d01y26-oh-v10-growing-mE16-full
- W&B path:
hazy-research/mlp-mixer/m05d01y26-oh-v10-growing-mE16-full - Training run name:
m05d01y26-online-hebbian-v10-growing-noSiLU-initSmall-120m-lr=0.0005-layers=12-dmodel=768-heads=12-initstd=0.02-miF=32-mEToks=64-memexpansion=16 - Source checkpoint on Perlmutter:
/pscratch/sd/j/jwl50/online-mlps/artifacts/checkpoints/mlp-mixer-pile/m05d01y26-online-hebbian-v10-growing-noSiLU-initSmall-120m-lr=0.0005-layers=12-dmodel=768-heads=12-initstd=0.02-miF=32-mEToks=64-memexpansion=16/last.ckpt - Hydra experiment:
mlp_mixer/online_hebbian_v10_noSiLU_initSmall_growing_120m
Files
lightning/last.ckpt: original PyTorch Lightning checkpoint.training_config.yaml: exact saved Hydra config from the training run.training_config_composed*.yaml: composed export configs retained for reproducibility.wandb_summary.json,wandb_config.json,wandb_metadata.json: W&B metadata captured at export time.
Latest Logged Metrics
| Metric | Value |
|---|---|
trainer/global_step |
16299 |
train/num-tokens_step |
8545894400 |
train/ppl_step |
12.028423824457565 |
val/ppl |
10.840018093394374 |
val/loss |
2.3832449913024902 |
val/num-tokens |
6133383168 |
Loading
Use the HazyResearch/mlp-mixer codebase with the Online Hebbian v10 growing mixer available. The checkpoint is the Lightning checkpoint, so instantiate the model from training_config.yaml / the Hydra experiment and load lightning/last.ckpt with the repo checkpoint-loading utilities.
The portable eval scripts can evaluate the same file under different active-feature schedules; the experiment-note labels correspond to 32->128, 32->256, and 32->528 linear schedules.